{"id":"397b4844-536f-4644-a11c-cd42a8495a4e","arxiv_id":"2506.11659","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A natural-language log retrieval pipeline using LLM-generated video and signal descriptions shows that prompt and model choice matter, but its proposed reliability metrics are not validated against any ground truth.","lead":"The paper tests an LLM-based system that turns test-drive videos and sensor logs into text descriptions, letting engineers search those logs with natural language. It compares prompts, models, and data modalities on the open Zenseact dataset, and introduces distribution-based metrics that are meant to show how reliable a search result is.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reliability metrics are never validated against ground truth, so the central claim that distribution shape indicates query reliability is unsupported; a labeled-relevance study is needed.","rationale":"The reader's weakest-assumption analysis identifies exactly the point on which the paper's central claim depends: the proposed metrics and distribution-shape criteria are asserted to indicate retrieval reliability without any ground-truth validation. I agree with that assessment. The paper is transparent about the lack of labels and about the manual nature of verification, and the API and qualitative examples are useful engineering contributions, but they do not establish the causal link between distribution shape and correctness. Because the same unlabeled similarity distributions are used both to choose the best prompt/model and to demonstrate reliability, the evaluation is circular. A labeled-relevance study, even on a subset of queries and records, would break the circularity and either support or refute the reliability claim. The internal inconsistency in Table 3 regarding whether values are similarities or distances is a concrete symptom of the same underlying problem: the metrics are not precisely tied to an evaluable notion of retrieval correctness. This does not change the reader's conditional verdict; it confirms that major revision with ground-truth validation and a baseline comparison is required before the central claims can be accepted.","tokens_in":14639,"tokens_out":4293,"duration_ms":42916,"concrete_test":"For each of the nine scenarios, with a fixed configuration (e.g., Prompt 4 and LLaVA), have two or more independent annotators label the top-50 retrieved records plus a random sample of 100 additional records as relevant or irrelevant, using both video frames and signal values; compute inter-annotator agreement. Then calculate, across scenarios, the rank correlation between the proposed reliability indicators (LGap, RLGap, Range, StdDev, and presence of a >0.9 cluster / <0.4 tail) and actual retrieval quality (precision@k, recall, NDCG). If the indicators do not correlate positively with labeled relevance, the RQ3 reliability claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is introduced in Section 4.3: that LGap, Range, RLGap, and the presence of a >0.9 top cluster and a <0.4 tail are valid indicators of whether retrieved records actually match the query. This assumption is never tested. The paper states in Section 1 that the data lack labels, then substitutes distribution-shape criteria for direct evaluation, but it never shows that a large gap, wide range, or S-shaped distance curve correlates with truly relevant retrieved records. The same unlabeled similarity distributions are used both to select Prompt 4 and LLaVA in Section 5.1 and to claim reliable retrieval in RQ3, making the evaluation circular. There is no baseline comparison (SQL, BM25, or a human-labeled relevance set), no precision/recall or NDCG computation, and no inter-rater agreement. A secondary internal inconsistency compounds this: Table 3 reports MaxD values above 1.0 (e.g., IV-4 MaxD = 1.0317) even though the text and figures describe cosine similarity scores above 0.9 as highly relevant, so it is unclear whether D_n is cosine similarity, cosine distance (1 - similarity), or another quantity. If the distribution-shape assumption is false, the headline conclusions about reliability, Prompt 4, and LLaVA collapse.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents an LLM-based retrieval pipeline for searching autonomous driving test logs. For each of the 1,473 Zenseact Open Dataset sequences, it generates textual descriptions from video (via LLaVA, InternVideo2, or CogVLM2) and from numeric signal tables via automated interpreters; a user's natural-language scenario query is embedded with all-MiniLM-L6-v2 and compared by cosine similarity against each record's combined description, with the video and signal scores summed. The paper evaluates six prompts, image versus video modality, and three video models across nine scenarios, using violin plots and six distribution-shape metrics (LGap, MinD, MaxD, Range, StdDev, RLGap) to judge retrieval reliability. It reports that Prompt 4 and LLaVA give the most reliable results and that these metrics are effective for assessing query outcomes.","tokens_in":14897,"tokens_out":6068,"duration_ms":53925,"significance":"If the central claims were supported, the work would be a practical contribution to natural-language log retrieval for autonomous driving: it uses a public dataset, a reproducible pipeline, and a concrete API for querying and visualization, and the comparison of prompts and models is informative. However, the evaluation never measures retrieval correctness against ground truth, the reliability construct is defined by the same distribution statistics used to select the winning configuration, and no baseline or efficiency measurement is reported. The current evidence does not establish the headline efficiency and reliability claims, although the gap is addressable with additional evaluation.","major_comments":[{"comment":"The claim that LGap, MinD, MaxD, Range, StdDev, and RLGap are reliable indicators of query correctness is not validated against any ground truth. The paper explicitly states that the data lack labels (Section 1) and substitutes distribution-shape criteria for direct evaluation, but it never demonstrates that a large gap, wide range, or S-shaped distance curve actually corresponds to retrieved records that match the query. No precision/recall, NDCG, baseline (SQL, BM25), or inter-rater agreement is reported, and the qualitative checks in Figures 6 and 7 cover selected examples only. Consequently, the conclusion in Section 8 that these metrics 'proved to be effective' is unsupported. A labeled subset with manual relevance judgments and a baseline comparison is needed.","section":"Section 4.3, Section 5.3, Section 8"},{"comment":"The evaluation is circular. Prompt 4 and LLaVA are selected because their cosine-similarity violin plots show long tails, large gaps, and well-distributed middle ranges; Section 5.3 then uses exactly those same distribution shapes and the Section 4.3 thresholds (scores above 0.9 and below 0.4) as evidence of reliable retrieval. Because the selection criterion and the success criterion are the same unlabeled distance distributions, the comparison cannot independently support the reliability claim. A concrete test would be to pre-register the distribution-shape criteria and validate them on held-out queries with known relevant records.","section":"Section 5.1.1, Section 5.1.3, Section 5.3"},{"comment":"The quantity D_n is not defined consistently. Table 3 reports MaxD values above 1 (e.g., IV-4 MaxD = 1.0317 and IV-6 MaxD = 1.0056), while Figure 4 labels the y-axis as cosine similarity and Section 4.3 defines reliable results using similarity thresholds of 0.9 and 0.4. If D_n is cosine distance (1 - similarity), then a MinD of 0.2345 (V-4) corresponds to similarity 0.7655, which contradicts the requirement of >0.9 in Section 4.3; if D_n is cosine similarity, values exceeding 1.0 are impossible. The definition of G_n in Section 4.3 also requires distances to be sorted, which is not stated. These issues affect the interpretation of every metric in Table 3.","section":"Table 3, Figure 4, Section 4.3"},{"comment":"The claim of 'improved efficiency and reliability' is not measured. No SQL baseline, no timing data, no query-composition cost, and no user-effort measurement are reported anywhere in Sections 4 or 5, and the only validation is the qualitative manual inspection of selected frames in Section 5.2. If the paper keeps the efficiency claim, it needs an empirical comparison (e.g., time or number of interactions to construct a query in SQL versus natural language); if only retrieval quality is claimed, the abstract and conclusions should be revised accordingly.","section":"Abstract, Section 6"}],"minor_comments":[{"comment":"The text says 'Driver consists of 29 multi-minute sequences'; this should read 'Drives'.","section":"Section 3.1"},{"comment":"The model listed as 'Cogvlm2-llama3-caption' is cited to reference [19], which is CogVideoX; please verify the citation and model name.","section":"Section 4.1.3"},{"comment":"The definitions of LGap and RLGap depend on consecutive differences G_n, which are only meaningful if the D_n values are sorted; please state this explicitly.","section":"Section 4.3"},{"comment":"The term 'cost-efficient' is asserted for Prompt 4 without reporting cost or runtime data; please add the measurement or soften the claim.","section":"Section 5.1.1"},{"comment":"The relationship between the violin plots (described in terms of similarity) and the table (which reports distances) should be stated explicitly, and the y-axis label should be aligned with the definition of D_n.","section":"Figure 4, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for EASE and the empirical setup is transparent about the lack of labels, but the gap between the claims and the evidence is substantial. I would encourage the editor to require a labeled or human-judged evaluation and a baseline comparison before publication, since the current evaluation is circular with respect to the paper's central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering extension of the authors' earlier work, scaling from 80 to 1,473 sequences and from single frames to full video, all on a public dataset. The prompt/model/modality comparisons are informative, and the qualitative observation that verbose multi-run prompts can hurt retrieval because long descriptions dilute short queries is plausible and actionable. But the headline claims about improved reliability and efficiency are not actually supported. There is no baseline (no SQL, no BM25, no human relevance labels), and the LGap/RLGap metrics are asserted as reliability indicators without ever being checked against ground truth. The same distance distributions are used to pick Prompt 4 and LLaVA in RQ1 and then to claim reliable retrieval in RQ3, which is circular.\n\nThe most concrete flaw: Section 4.3 defines the reliable-result criteria (a few similarity scores above 0.9, some below 0.4, no distinct middle clusters), but there is zero evidence that these distribution shapes correspond to true matches. The paper itself admits the data are unlabeled and that records can be described from multiple perspectives. Under those conditions, the distribution-shape story could easily be an artifact of description length or prompt verbosity, not a signal of retrieval quality. A small human-labeled study on even a few dozen queries would settle it.\n\nThere is also a technical inconsistency worth fixing: Table 3 reports MaxD values like 1.0317 for IV-4, which is impossible if these are cosine similarities (upper bound 1.0). Figure 4's y-axis says 'cosine similarity score,' but the narrative treats D_n as a distance. Section 4.3 defines D_n as distance, while the reliability criteria speak of similarity scores. That conflation needs to be cleaned up before the metrics can be trusted.\n\nWhat is genuinely useful: the paper ships a concrete pipeline on a public dataset, compares six prompts and three video LLMs in a realistic setting, and identifies a real failure mode for short-query/long-description retrieval. It is not a methods paper; the metrics are simple descriptive statistics.\n\nWho this is for: practitioners building log-retrieval tools for autonomous driving testing and researchers doing empirical LLM-based retrieval in unlabeled settings. Worth a serious referee, but not as-is. Major revision: add a labeled-relevance evaluation (even a small one), compare against at least one baseline (SQL for signal-only cases or a lexical retriever), fix the cosine/distance confusion, and report per-query variability. If the authors do that, this becomes a legitimately useful empirical contribution.","headline":"A useful engineering extension with informative prompt/model comparisons, but the reliability claims rest on an untested assumption about distance-distribution shape.","tokens_in":15438,"tokens_out":2035,"would_cite":false,"duration_ms":19638,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pipeline of LLM-written video and signal descriptions lets engineers search test logs in plain English and judge result reliability from score gaps.","keywords":["log retrieval","autonomous driving","LLM","natural language query","video description","signal data","cosine similarity","retrieval reliability"],"falsifier":"Choose a set of queries from the open dataset with human-annotated correct matches, compute the six metrics for each query, and check whether queries with large LGap, high RLGap, and wide range retrieve the correct records at the top; the reliability claim fails if any high-gap query returns unrelated records or if low-variance queries still retrieve correct ones.","tokens_in":14416,"feed_emoji":"🚗","tokens_out":7885,"duration_ms":70570,"temperature":0.7,"pith_summary":"The paper tries to establish that LLM-based retrieval can open up autonomous-driving test logs to natural-language queries, removing the need to know long signal names and SQL. It builds a two-modality pipeline: signal data is converted to text by fixed interpreters, video frames are converted to text by a video-language model, and a sentence-embedding model matches an engineer's query to the stored descriptions. Because log data has no ground-truth labels for what counts as a correct match, the paper proposes that query reliability can be read from the distribution of similarity scores across all records, gaps, range, and relative largest gap, and it validates this idea empirically on an open dataset of 1,473 driving sequences. If the approach holds, engineers can locate specific driving scenarios in plain English and gauge whether the results can be trusted without manual verification of every row. The paper also claims that concise prompts and video input are important for achieving discriminative descriptions.","feed_headline":"Plain English replaces SQL for searching driving logs","feed_subtitle":"LLM-written video and signal descriptions plus gap metrics show which search hits to trust.","key_machinery":"The scenario distance graph: for each query, the pipeline computes the cosine similarity between the query embedding and every record's combined (signal-plus-video) text description, sorts those distances $D_1 \\le D_2 \\le \\dots \\le D_{1473}$, and examines the consecutive gaps $G_n = D_{n+1} - D_n$. The six named metrics, LGap, MinD, MaxD, Range, StdDev, and RLGap, are all derived from this sorted distance distribution, and the paper defines a reliable result as one where a few records score above 0.9, some score below 0.4, and the rest spread out without forming clusters. This distribution-shape criterion is the mechanism that lets the paper evaluate retrieval quality in the absence of labels.","core_discovery":"The paper claims that natural-language scenario search over autonomous-driving logs is feasible without SQL by converting both signal logs and video frames into text descriptions, signals via automated interpreters and video via a video-language LLM, then ranking records by cosine similarity of a sentence-embedding model. Its central empirical conclusion is that this combined, text-based pipeline retrieves relevant scenarios, that video descriptions contribute motion information unavailable from a single image, and that the reliability of a query can be judged without ground truth by the shape of the sorted similarity-distance distribution: large gaps, wide range, high standard deviation, and high relative largest gap signal a successful, discriminative search, whereas low variance signals a failed search.","pith_inferences":["If the distribution-shape reliability criterion is accepted, the same metrics could transfer to other unlabeled retrieval settings, such as code search, incident reports, or regression-test selection, where precision cannot be computed directly, though they would first need validation against ground truth.","A direct test is available: run the pipeline on a subset of the open dataset with human-annotated query-relevance labels and check whether high LGap, high RLGap, and wide range actually predict higher precision at the top of the ranking.","The paper's finding that queries whose keywords appear in descriptions with the wrong semantics, for example 'no pedestrians' matching a query for 'pedestrians', produce false positives points to a retrievable-but-not-semantically-aligned failure mode; a negation-aware check could be added to the pipeline.","Because the current pipeline stores all descriptions and does brute-force comparison, applying it at industrial scale would require an embedding index and keyword pre-filtering; the proposed metrics could then double as a query-level confidence signal."],"forward_implications":["Engineers can replace SQL and detailed schema knowledge with natural-language scenario descriptions, because the pipeline converts both signal data and video into text before matching.","Query reliability can be assessed without labeled ground truth: a wide range, a large absolute and relative largest gap, and an S-shaped distribution of similarity scores are treated as indicators of a successful match, while low variance indicates failure.","Combining video with signal data adds motion-related information, such as slowing down or approaching intersections, that a single still frame cannot provide, making descriptions more discriminative.","Prompt formatting matters: a concise, structured prompt produced more reliable retrieval than longer multi-run prompts, which generated over-detailed descriptions that inflated similarity across all records.","Among the tested configurations, a video-capable vision-language model with the concise prompt gave the most reliable descriptions, supporting that configuration for practical use."],"supporting_citations":[{"why":"Supplies the open paired video-and-signal dataset that defines the empirical setting and all record counts in the study.","marker":"[3]"},{"why":"The prior single-frame, 80-record approach that this paper extends to full video and roughly twenty times more data.","marker":"[7]"},{"why":"The video-language model that produced the most reliable descriptions, carrying the central prompt and modality comparison.","marker":"[23]"},{"why":"One of the two alternative video-description models compared against the chosen model in the evaluation.","marker":"[18]"},{"why":"The other alternative video-description model compared in the evaluation, providing the baseline for model selection.","marker":"[19]"}],"fun_headline_variants":["LLM replaces SQL for driving log scenario search","Natural-language queries unlock driving logs, no SQL","Video-aware LLM retrieval with gap-based trust scores","Similarity gaps show when log searches hit or miss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes, without ground-truth validation, that the shape of the cosine-similarity distribution over all records, a few scores above 0.9, some below 0.4, and a spread in between, indicates that the query actually found the right records.","fun_headline_variants_meta":{"raw":{"variants":["LLM replaces SQL for driving log scenario search","Natural-language queries unlock driving logs, no SQL","Video-aware LLM retrieval with gap-based trust scores","Similarity gaps show when log searches hit or miss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000861,"raw_usage":{"total_tokens":3701,"prompt_tokens":874,"completion_tokens":2827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":2778}},"tokens_in":490,"tokens_out":2827,"duration_ms":20943,"temperature":1.0,"reasoning_tokens":2778,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:30.627517+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Choose a set of queries from the open dataset with human-annotated correct matches, compute the six metrics for each query, and check whether queries with large LGap, high RLGap, and wide range retrieve the correct records at the top; the reliability claim fails if any high-gap query returns unrelated records or if low-variance queries still retrieve correct ones.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior single-frame, 80-record approach that this paper extends to full video and roughly twenty times more data."}],"review_version":1}