Pith. sign in

REVIEW 4 major objections 5 minor 32 references

VLN-AVP: Zero-Shot Vision-Language Navigation with Hybrid Long-Short-Term Memory for Autonomous Valet Parking

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Pith's one-line read This paper claims that autonomous valet parking can be performed zero-shot in unseen underground garages by pairing a bird's-eye-view trajectory generator with a vision-language model that reads signs and follows natural-language instructio

desk verdict The benchmark and architecture are the real contributions; the headline performance numbers need error bars before they can be trusted. read the letter →

arxiv 2607.17767 v1 pith:G5E4D5AP submitted 2026-07-20 cs.RO

classification cs.RO
keywords vision-languagenavigationautonomousvaletparkingzero-shotbird's-eye-viewperceptionlong-termmemoryshort-termundergroundgaragebenchmarknaturallanguageinstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that autonomous valet parking does not require pre-built maps or predefined slot numbers. By combining a high-frequency bird's-eye-view perception module that generates drivable candidate trajectories with a vision-language model that interprets signs and user commands, the system can navigate unfamiliar garages from natural-language instructions. To handle the VLM's low-frequency, single-frame reasoning, a short-term memory retains recent sign detections, while a long-term topological memory accumulates successful routes so repeated visits become faster and cheaper. The paper also contributes a simulation dataset of ten underground garages with over 1,000 navigation episodes, the first vision-language navigation benchmark for parking, and reports success rates of 65.2% on regular tasks, 45.4% on challenging tasks, and 63.3% in real-vehicle trials.

What carries the argument

The load-bearing mechanism is the hybrid long-short-term memory system feeding a hierarchical decision module. The short-term memory is a queue of recent sign detections with timestamps, bounding boxes, and vehicle yaw, filtered to keep only relevant, current, same-direction cues; it stabilizes VLM decisions when no sign is visible in the current frame. The long-term memory is a directed topological graph of junction nodes, landmark nodes, and direction-vector edges, updated by user feedback and reused on later episodes. The hierarchical decision module prefers a stored graph route when one exists, falls back on the cached VLM action otherwise, and maps the chosen action onto a kinematically

What would settle it

Measure, across the Regular benchmark episodes, how often the generated candidate trajectory set contains at least one trajectory whose endpoint lies within the target area. If that frequency is materially below the reported 65.2% success rate, the bottleneck is BEV perception, not the VLM or memory modules; if the frequency is high but success is low, the bottleneck lies in decision-making.

Watch

Extended reading notes

Core claim

The central claim is that the combination of kinematically filtered candidate trajectories from BEV lane perception with a low-frequency VLM decision-maker, plus two complementary memories, produces zero-shot vision-language navigation in underground parking garages. The VLM is not asked to plan in metric space; it chooses among a small set of drivable trajectories derived from predicted lane centerlines, and the memories let it act on cues absent from the current frame. The paper demonstrates that this architecture outperforms both vision-language navigation baselines and VLM-based driving baselines, that removing either memory degrades success, and that the long-term memory reduces VLM cal

Load-bearing premise

The whole pipeline rests on the BEV lane-detection model, trained on open-road data, producing a correct lane centerline in an unfamiliar garage, because the candidate trajectories are generated from those lane lines and the VLM can only choose among them.

Editorial extensions

If this is right

  • Valet parking can be initiated in an unseen garage from a natural-language instruction such as 'park near the elevator' without a pre-built HD map.
  • A vehicle can safely follow a VLM's high-level commands by restricting its choices to kinematically feasible candidate trajectories, avoiding the frequent U-turns that make typical VLN policies impractical on real cars.
  • Repeated trips to the same garage build a lightweight topological graph that raises success rate and cuts VLM API calls and cost by about 75%.
  • The new benchmark, with ten photorealistic garage scenes and over 1,000 episodes, provides a standardized testbed for evaluating vision-language navigation in underground parking.
  • Caching recent sign detections in short-term memory lets the VLM make consistent decisions even when the current image contains no visible navigation cue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The short-term-cue memory idea could transfer to other map-free indoor navigation settings, such as delivery robots or warehouse vehicles, where signage is the dominant navigation cue.
  • The long-term topological graph, stored as direction vectors rather than steering commands, is naturally shareable across a fleet of vehicles; pooled successful routes could form a collective map-free memory.
  • The directional filter that drops remembered signs when the vehicle turns assumes sign relevance is tied to yaw; multi-level or looped garages may require a position-aware or lane-aware filter.
  • If a cheaper or faster VLM were substituted, the memory components might partially compensate, and the challenging benchmark subset, with composite goals like 'EV charger near the exit', would directly stress-test that substitution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VLN-AVP, a zero-shot vision-language navigation framework for autonomous valet parking. The system replaces pre-built HD maps with MapTR BEV lane-centerline perception, uses a VLM (Gemini-3-flash) for high-level decisions, and adds a hybrid memory: a short-term perception memory based on Grounding DINO detections and a long-term topological memory with validity bits and graph-based retrieval. A hierarchical decision module selects among RS-curve candidate trajectories. The authors also introduce a 3DGS-based dataset of 10 underground garages with over 1,000 VLN episodes and report simulation and real-world experiments against VLN and AD baselines, including ablations. The headline claims are an over 25% SR improvement over VLN methods, over 15% over AD methods, and 63.3% SR in 30 real-world trials.

Significance. If substantiated, the paper would make a useful contribution: it is the first VLN benchmark for underground parking, it demonstrates a map-free zero-shot parking navigation pipeline with human-in-the-loop correction, and the hybrid memory reduces VLM API calls and cost. The system design is modular and mostly reproducible from existing components. However, the central empirical claim is a quantitative performance advantage, and the evidence currently consists of single-point estimates without variance, confidence intervals, significance tests, or full protocol details. The contribution is therefore promising but not yet established at the level claimed in the abstract and conclusions.

major comments (4)
  1. [Abstract; Tables III, IV, V; Fig. 8] All reported success-related metrics are single-point estimates. No number of repeated runs/seeds, standard deviations, confidence intervals, or significance tests are given. The headline 'over 25% improvement' and 'over 15% improvement' are thus not statistically supported. In Table IV, for example, the difference between Ours (63.3%) and Ours w/o LTM (53.3%) on 30 trials is exactly 3 episodes, which is within plausible sampling noise. Please add repeated evaluations, per-episode counts, confidence intervals, and at least a basic significance test for the main comparisons.
  2. [Sec. IV-A, IV-B, V] The evaluation protocol is underspecified. Please state the number of episodes in the Regular and Challenging subsets, per-scene episode counts, the exact success criterion (e.g., stopping within the target bounding box, distance threshold), how instructions and targets are sampled, and whether all methods run on identical episode sets. In addition, the dataset is not currently released: Sec. V says the authors 'plan to further improve and publicly release' it, so the benchmark cannot yet be independently used by the community.
  3. [Sec. III-A, Eq. (1)] The method's entire action space T_t is generated from MapTR lane-centerline predictions in unseen underground garages. The paper provides no quantitative evaluation of MapTR on parking-garage scenes. If lane centerlines are sparse, disconnected, or geometrically wrong, the candidate trajectory set cannot contain a drivable path toward the requested target, and neither the VLM nor the memory modules can recover. Please report MapTR's lane extraction quality on the VLN-AVP scenes (e.g., centerline recall, trajectory reachability rate to target locations) and discuss how the system handles cases where no candidate trajectory leads to the target.
  4. [Sec. III-B, III-D, IV-E; Table VI; Fig. 8] The ablations attribute the performance gains to the memory modules, but they inherit the statistical issue. Figure 8, which is the main evidence for LTM's 'learning ability', has no error bars or aggregation details (each point is a single episode? a window? a single scene?), and its axes are unlabeled. Moreover, the memory system depends on several free parameters introduced in Secs. III-B and III-D (K_STM, delta_theta_max, delta_t_max, theta_threshold) whose values are not reported and for which no sensitivity analysis is provided. Please add robustness experiments or justify why the chosen values are not the result of tuning that favors the proposed method.
minor comments (5)
  1. [Sec. III-B, Eq. (4)] Eq. (4) writes the prompt as [I_front, {I_detect,j}, S, g], but the STM frame defined in Sec. III-B is m = (I_detect, B, t, theta). Clarify which components of the STM frame are actually fed to the VLM.
  2. [Sec. IV-B, Ref. [32]] The simulator is described as being built on the Real2Sim2Real simulator in REAP [32], which is an author preprint. Please specify whether the simulator code will be released, or provide enough implementation documentation for others to reproduce the benchmark.
  3. [Table II] The 'Render Method' column lists '3DGS' for all datasets, which is ambiguous. Clarify how each existing dataset was rendered and whether the VLN-AVP scenes use 3DGS exclusively.
  4. [Sec. IV-E, Fig. 8] Add axis labels, units, and a legend to Figure 8. Also state how many episodes each plotted point aggregates and whether the curve is over a single scene or multiple scenes.
  5. [Sec. IV-D, Table IV] The real-world comparison includes only LaMPilot. State why other AD baselines were omitted, and describe the real vehicle's sensor setup and the source of ground-truth localization/target positions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the system is evaluated on an independently constructed benchmark and the only self-citation is simulator infrastructure.

full rationale

The paper's contribution is a modular system integration rather than a derived mathematical law: MapTR produces candidate trajectories T_t = f_optim(L_t), the VLM converts instructions into sub-goals and meta-actions, and the short/long-term memories select among candidate trajectories. None of these modules is fitted to the reported success rates, and SR/OSR/SPL are measured on the constructed benchmark rather than derived from the method's own equations. Baselines receive the same BEV perception and the same VLM core, so the comparison isolates the decision and memory components; this is a fair-comparison design, not a fitted-input-called-prediction loop. The only apparent self-citation is REAP [32], used to describe the customized simulator's design; it is shared infrastructure from the same group (Changze Li) but does not encode the target results or any success-rate numbers. The lack of error bars or significance tests is a statistical-reliability concern, not circularity, because the reported numbers are not equivalent to the method's inputs by construction. No circular step is present.

Assumptions & free parameters 4 free parameters · 6 assumptions · 2 invented entities

The work is empirical, so the ledger is dominated by domain assumptions about pretrained component transfer (MapTR, Grounding DINO, Gemini-3-flash), simulator fidelity, and hand-chosen memory thresholds. The only genuinely new structures are the STM and LTM modules, which are not independently verifiable outside the paper. No speculative physical entities are introduced.

free parameters (4)
  • K_STM (short-term memory cache size) = 6
    Hand-chosen cache size for the STM queue; no sensitivity analysis is reported (Sec. IV-B).
  • Delta theta_max (directional filter yaw threshold) = not reported
    Threshold for removing STM frames with yaw angles significantly different from the current pose (Sec. III-B).
  • Delta t_max (temporal filter age threshold) = not reported
    Threshold for removing STM frames older than this duration (Sec. III-B).
  • theta_threshold (LTM trajectory cosine-similarity threshold) = not reported
    Threshold for matching a candidate trajectory to the LTM edge direction during graph-based selection (Sec. III-E).
assumptions (6)
  • domain assumption MapTR trained for open-road HD-map construction transfers to underground parking without fine-tuning (Sec. III-A).
    The entire candidate-trajectory space is generated from MapTR centerline predictions in unseen garages; no parking-specific retraining or direct MapTR accuracy evaluation is reported.
  • domain assumption Grounding DINO with the single prompt 'sign' detects all navigation-critical signage in parking garages (Sec. III-B).
    STM contents and downstream VLM reasoning depend on these detections; missed signs cannot be recovered if they are not cached.
  • domain assumption Gemini-3-flash reliably maps prompt+images to the four actions {forward, left, right, stop} in the tested garages (Sec. III-C).
    The VLM is the sole semantic decision core; its reasoning is only audited qualitatively in two case studies, not quantitatively beyond the final metrics.
  • domain assumption 3D Gaussian Splatting reconstruction from handheld LiDAR/fisheye scans is a faithful proxy for real vehicle sensors and dynamics (Sec. IV-A.1).
    The primary quantitative evidence comes from simulation built on the REAP simulator; the real-world validation covers only 30 trials in 2 garages.
  • domain assumption Benchmark ground-truth trajectories and success boxes are correct and unambiguous (Sec. IV-A.2).
    The episode construction and annotation protocol are not described in detail, so the benchmark's ground truth cannot be independently checked.
  • ad hoc to paper LTM validity-bit update rules (v=1 on success, v=0 on user/system failure) correctly infer environment changes and target correctness from sparse feedback (Sec. III-D).
    These hand-authored rules are central to LTM learning; no analysis is given for how false positive/negative feedback propagates through the graph.
invented entities (2)
  • Short-Term Perception Memory (STM)
    purpose: Retains Grounding DINO sign detections across frames to compensate for the low-frequency VLM and single-frame reasoning (Sec. III-B).
    Supported only by internal ablations (Table VII); no external artifact or out-of-paper prediction is provided.
  • Long-Term Topological Memory (LTM)
    purpose: A topological graph of junction/landmark nodes and direction-vector edges that learns successful routes from past episodes and reduces VLM calls (Sec. III-D).
    Validated by internal Figure 8 and Table VI, with no released implementation; it also complicates the 'zero-shot' label for repeated-garage tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLN-AVP: Zero-Shot Vision-Language Navigation with Hybrid Long-Short-Term Memory for Autonomous Valet Parking." pith.science (2026). https://pith.science/paper/G5E4D5AP

@misc{pith2026260717767,
  author       = {Pith},
  title        = {Pith review of: VLN-AVP: Zero-Shot Vision-Language Navigation with Hybrid Long-Short-Term Memory for Autonomous Valet Parking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G5E4D5AP}},
  note         = {Machine review of arXiv:2607.17767}
}
read the original abstract

Existing methods in Autonomous Valet Parking (AVP) typically rely on pre-built maps, which severely restricts their scalability to unseen environments and open-vocabulary targets. Inspired by the application of Vision-Language Models (VLMs) in Vision-Language Navigation (VLN) tasks, we propose VLN-AVP, a zero-shot navigation framework for AVP tasks. By combining the precise spatial perception of a Bird's-Eye-View (BEV) model with the general intelligence of VLMs, our framework 1) eliminates the dependency on pre-built maps, 2) interprets semantic environmental contexts in parking scenarios, and 3) enables intuitive navigation following natural language instructions. Specifically, we introduce a hybrid memory system: a short-term perception memory tracks semantic visual cues to address the limitations of VLM's single-frame reasoning in existing methods, while a long-term topological memory facilitates stable policy learning from past experiences. To bridge the gap in existing benchmarks, we also present the VLN-AVP dataset and benchmark. Featuring 10 high-fidelity parking scenes and over 1,000 navigation episodes, it has the largest number of garage scenes to date and is the first VLN benchmark for underground parking. Extensive experiments demonstrate that in simulation, our method achieves an over 25% improvement in success rate compared to VLN methods and an over 15% improvement compared to other autonomous driving methods. Furthermore, it attains a leading success rate in real-world vehicle experiments, proving its practical feasibility.

Figures

Figures reproduced from arXiv: 2607.17767 by the authors.

Figure 1
Figure 1. Traditional AVP systems require pre-built high-definition (HD) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The structure of VLN-AVP. The orange block updates at high frequency to provide low-level BEV perception and candidate trajectories. The [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualized LTM on a simulated scene after 10 navigation episodes. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: 10 scenes in the VLN-AVP dataset. Top row: 3DGS renderings of 5 scenes; Bottom row: Top-down views of the remaining 5 scenes. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: A successful case study of our VLN-AVP system navigating a complex, multi-stage instruction in simulation. The left-most panel shows the global [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: A qualitative experiment on a real-world vehicle platform. The system initially navigates to an incorrect elevator hall but successfully corrects its [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Performance comparison over 50 navigation attempts on one scene. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 5 linked inside Pith

  1. [1]

    Autonomous valet parking system architecture,

    M. Chircaet al., “Autonomous valet parking system architecture,” in ITSC, 2015, pp. 2619–2624

  2. [2]

    Avp-slam: Semantic visual mapping and localization for autonomous vehicles in the parking lot,

    T. Qinet al., “Avp-slam: Semantic visual mapping and localization for autonomous vehicles in the parking lot,” inIROS, 2020, pp. 5939– 5945

  3. [3]

    Less-map: Lightweight and evolving semantic map in parking lots for long-term self-localization,

    M. Liuet al., “Less-map: Lightweight and evolving semantic map in parking lots for long-term self-localization,” inICRA, 2024, pp. 17 990–17 996

  4. [4]

    Drivevlm: The convergence of autonomous driving and large vision-language models,

    X. Tianet al., “Drivevlm: The convergence of autonomous driving and large vision-language models,”arXiv:2402.12289, 2024

  5. [5]

    Drivelm: Driving with graph visual question answer- ing,

    C. Simaet al., “Drivelm: Driving with graph visual question answer- ing,” inECCV, 2024, pp. 256–274

  6. [6]

    Lampilot: An open benchmark dataset for autonomous driving with language model programs,

    Y . Maet al., “Lampilot: An open benchmark dataset for autonomous driving with language model programs,” inCVPR, 2024, pp. 15 141– 15 151

  7. [7]

    Dilu: A knowledge-driven approach to autonomous driving with large language models,

    L. Wenet al., “Dilu: A knowledge-driven approach to autonomous driving with large language models,” inICLR, 2024

  8. [8]

    Vision-and-language navigation: A survey of tasks, methods, and future directions,

    J. Guet al., “Vision-and-language navigation: A survey of tasks, methods, and future directions,”ACL, pp. 7606–7623, 2022

Show all 32 references
  1. [9]

    Automated valet parking and charging for e- mobility,

    U. Schwesingeret al., “Automated valet parking and charging for e- mobility,” inIV, 2016, pp. 157–164

  2. [10]

    Inverse perspective mapping-based neural occupancy grid map for visual parking,

    X. Muet al., “Inverse perspective mapping-based neural occupancy grid map for visual parking,” inICRA, 2023, pp. 8400–8406

  3. [11]

    Avp-loc: Surround view localization and relocaliza- tion based on hd vector map for automated valet parking,

    C. Zhanget al., “Avp-loc: Surround view localization and relocaliza- tion based on hd vector map for automated valet parking,” inIROS, 2021, pp. 5552–5559

  4. [12]

    Think global, act local: Dual-scale graph transformer for vision-and-language navigation,

    S. Chenet al., “Think global, act local: Dual-scale graph transformer for vision-and-language navigation,” inCVPR, 2022, pp. 16 537– 16 547

  5. [13]

    Navid: Video-based vlm plans the next step for vision- and-language navigation,

    J. Zhanget al., “Navid: Video-based vlm plans the next step for vision- and-language navigation,” inRSS, 2024

  6. [14]

    Navila: Legged robot vision-language-action model for navigation,

    A. Chenget al., “Navila: Legged robot vision-language-action model for navigation,” inRSS, 2025

  7. [15]

    Navgpt: Explicit reasoning in vision-and-language navigation with large language models,

    G. Zhouet al., “Navgpt: Explicit reasoning in vision-and-language navigation with large language models,” inAAAI, 2024, pp. 7641– 7649

  8. [16]

    Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation,

    J. Chenet al., “Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation,” inACL, 2024, pp. 9796– 9810

  9. [17]

    Apexnav: An adaptive exploration strategy for zero-shot object navigation with target-centric semantic fusion,

    M. Zhanget al., “Apexnav: An adaptive exploration strategy for zero-shot object navigation with target-centric semantic fusion,”IEEE Robotics and Automation Letters, 2025

  10. [18]

    Human-like navigation in a world built for humans,

    B. Chandakaet al., “Human-like navigation in a world built for humans,” inCoRL, 2025

  11. [19]

    Gpt-driver: Learning to drive with gpt,

    J. Maoet al., “Gpt-driver: Learning to drive with gpt,” arXiv:2310.01415, 2023

  12. [20]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model,

    Z. Xuet al., “Drivegpt4: Interpretable end-to-end autonomous driving via large language model,”RA-L, 2024

  13. [21]

    Orion: A holistic end-to-end autonomous driving framework by vision-language instructed action generation,

    H. Fuet al., “Orion: A holistic end-to-end autonomous driving framework by vision-language instructed action generation,” inICCV, 2025, pp. 24 823–24 834

  14. [22]

    Lightemma: Lightweight end-to-end multimodal model for autonomous driving,

    Z. Qiaoet al., “Lightemma: Lightweight end-to-end multimodal model for autonomous driving,”arXiv:2505.00284, 2025

  15. [23]

    Drive like a human: Rethinking autonomous driving with large language models,

    D. Fuet al., “Drive like a human: Rethinking autonomous driving with large language models,” inWACVW, 2024, pp. 910–919

  16. [24]

    Maptr: Structured modeling and learning for online vectorized hd map construction,

    B. Liaoet al., “Maptr: Structured modeling and learning for online vectorized hd map construction,” inICLR, 2023

  17. [25]

    Optimal paths for a car that goes both forwards and backwards,

    J. Reeds and L. Shepp, “Optimal paths for a car that goes both forwards and backwards,”Pacific journal of mathematics, vol. 145, no. 2, pp. 367–393, 1990

  18. [26]

    Grounding dino: Marrying dino with grounded pre- training for open-set object detection,

    S. Liuet al., “Grounding dino: Marrying dino with grounded pre- training for open-set object detection,” inECCV, 2024, pp. 38–55

  19. [27]

    3d gaussian splatting for real-time radiance field rendering

    B. Kerblet al., “3d gaussian splatting for real-time radiance field rendering.”ACM Trans. Graph., vol. 42, no. 4, pp. 1–14, 2023, art. no. 139

  20. [28]

    Sups: A simulated underground parking scenario dataset for autonomous driving,

    J. Houet al., “Sups: A simulated underground parking scenario dataset for autonomous driving,” inITSC, 2022, pp. 2265–2271

  21. [29]

    Development of occupancy prediction algorithm for under- ground parking lots,

    S. Wang, “Development of occupancy prediction algorithm for under- ground parking lots,”arXiv:2409.00923, 2024

  22. [30]

    Letsgo: Large-scale garage modeling and rendering via lidar-assisted gaussian primitives,

    J. Cuiet al., “Letsgo: Large-scale garage modeling and rendering via lidar-assisted gaussian primitives,”ACM Trans. Graph., vol. 43, no. 6, pp. 1–18, 2024

  23. [31]

    A point cloud dataset and deep learning method for semantic segmentation of underground garages,

    H. Yueet al., “A point cloud dataset and deep learning method for semantic segmentation of underground garages,”CACAIE, vol. 40, no. 23, pp. 3726–3749, 2025

  24. [32]

    Reap: Reinforcement-learning end-to-end autonomous parking with gaussian splatting simulator for real2sim2real transfer,

    C. Liet al., “Reap: Reinforcement-learning end-to-end autonomous parking with gaussian splatting simulator for real2sim2real transfer,” arXiv preprint arXiv:2605.08713, 2026

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.