{"id":"e78df702-2620-4cc3-84f7-cef8cd28c18c","arxiv_id":"2501.07024","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A Retrieval-Augmented Generation system for archive search combines translation, query routing, hybrid BM25/vector retrieval, and LLM response generation, reporting 80.56% precision with Mistral on 110 synthetic queries.","lead":"This paper builds an AI search system for digital archives that turns images, audio, and video into text, then answers natural language queries by naming matching file IDs. It may interest archivists and searchers, but the evaluation lacks a conventional baseline and uses synthetic test data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All reported metrics are computed from file IDs the LLM mentions in its response, not from retriever outputs, so they measure answer grounding rather than retrieval; the claimed improvement over conventional search is therefore unsupported.","rationale":"The reader's identified weakest assumption is the decisive one: the evaluation protocol equates 'retrieved files' with file IDs mentioned in the LLM's generated response, so the reported metrics are not retrieval metrics. This directly undermines the abstract's claim of improved search precision and the conclusion that the proposed system advances archival search. The absence of an end-to-end conventional baseline removes the basis for the 'significant improvements over conventional approaches' assertion, and the synthetic templated queries with implicit relevance labels make the reported numbers even harder to interpret. My proposed check would separate generation-level citation correctness from retriever-level retrieval correctness by scoring actual top-k outputs and verifying that all cited IDs exist in the corpus. Since no such retriever-grounded evaluation is presented, the existing rejection verdict stands unchanged.","tokens_in":7970,"tokens_out":4717,"duration_ms":47068,"concrete_test":"Recompute Tables 1–3 using the actual top-k results of the hybrid retriever (e.g., k=10 and k=20) as the retrieved set, with the same filetype/topic relevance labels, and compare these retriever-based precision/recall/F1 values against the response-cited metrics. If the retriever-based values differ materially from the response-cited values, or if any response-cited file ID does not exist in the knowledge base, then the reported metrics conflate generation with retrieval and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 states: \"In our research, the retrieved files are those referenced in the LLM's response text, rather than files directly retrieved from the underlying retrieval system\" (Figure 5). Every precision/recall/F1/hit-rate number in Tables 1–3 and Figure 7 is therefore a measure of whether the LLM happens to cite correct file IDs in its generated text, not a measure of retrieval quality. Figure 5 makes the gap concrete: the retriever extracts four files, the response mentions two, and the \"retrieved files\" are redefined as the two mentioned IDs. This conflation is load-bearing because the abstract and conclusions claim improved search precision and relevance; the central quantitative evidence never isolates the retriever. No end-to-end conventional archive search baseline is reported; alpha=0/1 within the hybrid-retriever ablation is not the same as a traditional search system. The synthetic dataset of 110 templated queries (Section 4.1) with implicit filetype/topic relevance labels adds a second layer: if the LLM can infer plausible IDs from templates, precision can be high even when the retriever is uninformative. Without a retriever-output-based scoring pass or human-validated relevance judgments, the headline 80.56% average precision cannot be interpreted as retrieval performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based RAG framework for smart search in digital archive systems. The architecture includes AI-generated metadata, BGE-M3 embeddings, Pinecone indexing, a hybrid retriever that linearly blends BM25 and embedding scores via an alpha parameter, a query router, a translator, post-processors, and a response synthesizer. The authors evaluate the system through four experiments: comparing Mistral 7B, Synatra 7B, and Llama 2 7B; tuning the alpha parameter; testing English and Korean queries; and an ablation study. They report that Mistral achieves 80.56% average precision and conclude that the framework yields significant improvements over conventional approaches.","tokens_in":8204,"tokens_out":2276,"duration_ms":23869,"significance":"If the reported results were valid, the paper would demonstrate a practically useful RAG pipeline for archival search, and the authors deserve credit for releasing their experimental dataset and code and for describing the architecture in reproducible detail. However, the central quantitative claim is not supported by the evaluation as designed. The paper explicitly redefines 'retrieved files' as the file IDs mentioned in the LLM's generated response rather than the files returned by the retriever, so all reported precision, recall, F1, and hit-rate numbers measure answer grounding in synthetic files, not retrieval quality. In addition, no conventional keyword-only or BM25-only end-to-end baseline is reported, and the hybrid-retriever ablation uses the same 110 queries for both tuning alpha and reporting the optimal value. These issues are load-bearing because the abstract and conclusions claim significant improvements in search precision and relevance. The contribution is therefore better viewed as a system description with an illustrative demonstration than as an experimentally validated retrieval method.","major_comments":[{"comment":"The evaluation metrics are computed over files mentioned in the LLM's response, not files returned by the retriever. The paper states this explicitly in Section 4.3: 'the retrieved files are those referenced in the LLM's response text, rather than files directly retrieved from the underlying retrieval system.' Figure 5 illustrates the conflation: the retriever extracts four files, but the response mentions two, and those two are then treated as the retrieved files. Consequently, every precision, recall, F1, and hit-rate number in Tables 1-3 and Figure 7 measures whether the LLM happens to cite relevant synthetic file IDs in its generated text, not whether the hybrid retriever retrieved relevant files. Since the abstract and conclusions claim improvements in search precision and relevance, this redefinition invalidates the central quantitative claim.","section":"Section 4.3, Figure 5, Tables 1-3"},{"comment":"The alpha parameter in Eq. (1) is a free parameter, and Section 5.2 tunes it over values 0.0 to 1.0 on the same 110 queries used to report the final performance. The statement that alpha = 0.5 or 0.8 represents an 'optimal balance' is therefore circular with respect to the reported numbers: the same data are used both to select alpha and to demonstrate the benefit of that selection. The authors should use a held-out validation set or cross-validation when selecting alpha and report performance on a separate test set. Without this separation, the gains attributed to hybrid retrieval are not evidence of a general advantage.","section":"Section 5.2, Eq. (1)"},{"comment":"The evaluation dataset consists of LLM-generated synthetic files and 110 templated queries whose relevance labels are implicit in the filetype/topic construction. This setup conflates retrieval quality with the LLM's ability to infer plausible file IDs from templated prompts: if the model can guess a syntactically valid ID that matches the template, precision can be high even when the retriever is uninformative. There is also no human-validated relevance judgment set or standard IR benchmark. The reported 80.56% precision cannot be interpreted as retrieval performance without a retriever-output-based scoring pass and validation of the synthetic relevance labels on a real archive corpus.","section":"Section 4.1, Section 4.2"},{"comment":"The claim of 'significant improvements over conventional approaches' is never tested. No end-to-end baseline using keyword-only search, BM25-only retrieval, or a standard vector-only retrieval system is reported; the alpha = 0 and alpha = 1 rows in the hybrid-retriever ablation are not equivalent to a conventional archive search system because the response synthesizer and LLM-based answer formatting are still present. Without comparison to such baselines, the conclusion that the proposed framework advances archival search is unsupported.","section":"Abstract, Section 5.1, Section 6"}],"minor_comments":[{"comment":"The observation that even alpha values have lower execution times than odd alpha values is presented without explanation or error bars; this pattern appears likely to be an artifact of small sample size or measurement noise and should be supported with repeated runs and confidence intervals.","section":"Section 5.2, Figure 7b"},{"comment":"The arrows indicating increases and decreases in the ablation table are not defined in the caption or the surrounding text; a sentence explaining the notation would improve readability.","section":"Table 3"},{"comment":"There are several typos and formatting inconsistencies, including 'Comparision' in Section 4.2, 'A verage' in Table 1, and unformatted hyperlink URLs in the reference list; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper is not poorly written at the architectural level, and the authors are transparent about their evaluation choices, but the transparency exposes a fundamental mismatch between the claims and the measurements. Because the evaluation metric, the lack of baselines, and the alpha-tuning procedure all affect the headline result, the paper would require a substantially redesigned experimental study rather than a local revision. I therefore recommend rejection in its current form, while noting that a future version with retriever-based evaluation and proper baselines could be a reasonable contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim is not supported by the experiments. The paper reports 80.56% average precision for Mistral, but that number is computed from file IDs the LLM mentions in its response, not from what the retriever returns. The paper itself says this in Section 4.3 and illustrates it in Figure 5: the retriever extracts four files, the response mentions two, and those two are then treated as the \"retrieved files.\" That is a measure of answer grounding, not retrieval quality. And since there is no keyword-only or BM25-only baseline anywhere in the evaluation, the abstract's claim of \"significant improvements over conventional approaches\" is never actually tested.\n\nWhat the paper does well: the architecture is clearly described and uses sensible RAG components — translator, router, hybrid retriever with BM25 and BGE-M3 embeddings, reranker, and response synthesizer. The ablation study is a good idea, and reporting execution time is useful. They also make the dataset and code available on GitHub. For someone building an archive search product, this is a reasonable engineering blueprint.\n\nThe soft spots are load-bearing. First, the evaluation conflation: by defining \"retrieved files\" as the IDs in the LLM's answer, the metrics can be inflated by a model that guesses plausible IDs from the query template. The synthetic dataset of 110 templated queries with LLM-generated files means the relevance labels are weak. Second, the hybrid retrieval weight alpha is tuned on the same queries used for the reported results, making the \"optimal balance at 0.8\" claim circular. Third, the absence of any conventional baseline is fatal for the main contribution. The ablation shows removing the router hurts F1 by 38.73%, which is interesting, but it still doesn't tell us how this compares to a simple keyword search on the same data.\n\nThis looks like an honest engineering report, not a research result. The system description could be useful to practitioners, but the evaluation does not demonstrate improved retrieval. If the authors re-ran the experiments with retriever-output-based scoring, added BM25 and keyword baselines, and used human-validated relevance judgments, the paper would be worth another look. As it stands, I would not cite it for its performance claims, and I would not bring it to a reading group without a strong caveat about the metric issue.","headline":"Fails to support its headline claim: the reported metrics measure LLM answer grounding, not retrieval, and no conventional baseline is tested.","tokens_in":8746,"tokens_out":2073,"would_cite":false,"duration_ms":20260,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM-based RAG framework with a translator, router query engine, hybrid BM25-vector retrieval, post-processors, and response synthesizer improves digital archive search, with Mistral 7B citing relevant file IDs at…","keywords":["archive systems","smart search","retrieval-augmented generation","large language models","hybrid retrieval","BM25","multilingual search","AI-generated metadata"],"falsifier":"Have human annotators re-run the 110 queries and check whether every file ID the LLM cites exists in the archive and matches the requested file type and topic; if a substantial share of cited IDs are hallucinated or irrelevant, the 80.56% precision claim collapses. A second check would repeat the pipeline on a real archival corpus with real user queries and compare against human relevance judgments.","tokens_in":7744,"feed_emoji":"🔍","tokens_out":8785,"duration_ms":72736,"temperature":0.7,"pith_summary":"This paper proposes a smart search framework for digital archives that combines retrieval-augmented generation (RAG) with large language models to answer natural-language queries with structured responses that cite specific file IDs. The authors argue that this approach improves on conventional keyword-based search by adding semantic understanding, translating multilingual queries into English, and converting images, audio, and video into searchable text through AI-generated descriptions. Their experiments on 110 templated queries across four file types show Mistral 7B as the best-performing tested LLM, with 80.56% average precision in naming relevant file IDs and a hit rate of 93.64%. The ablation study shows that the translator and router are load-bearing: removing the router drops the F1 score by 38.73 percentage points.","feed_headline":"Mistral 7B archive search reaches 80.56% precision","feed_subtitle":"A hybrid BM25-vector retriever with query routing names the right files on 110 templated queries.","key_machinery":"The load-bearing mechanism is the RAG pipeline: the knowledge base stores every file as text, with AI-generated descriptions for non-textual media, plus vector embeddings; the translator converts queries into English and responses back into the user's language; the router query engine sends each query to the appropriate media-specific engine; the hybrid retriever combines BM25 keyword scores with vector embedding scores through the weighted sum $\\alpha \\cdot \\text{Embedding Score} + (1-\\alpha) \\cdot \\text{BM25 Score}$; reranking and long-context reordering refine the retrieved nodes; and the response synthesizer asks an LLM to produce a structured answer that names file IDs. The definition of \"retrieved files\" as the IDs appearing in the LLM's answer is what ties retrieval quality to the generated response.","core_discovery":"The central claim is that a RAG pipeline built for archives can retrieve and cite files by ID in generated answers, and that this design improves on conventional keyword search in precision and usefulness. The strongest evidence is Experiment 1, where Mistral 7B achieves 80.56% average precision, 93.64% hit rate, and an F1 of 66.21%, clearly ahead of Synatra and Llama 2. The paper also finds that the optimal hybrid retriever weighting sits near alpha 0.8, that English queries outperform Korean queries for every tested model, and that the translator and query router are essential components, since removing the router drops F1 by 38.73 percentage points.","pith_inferences":["The reported scores therefore measure citation reliability as much as retrieval: a retriever that returns the right files will still score low if the LLM omits or miscites file IDs.","Because the 110 queries are templated, real-world archive queries with more varied wording and intent would likely show lower precision; adding paraphrased or longer queries would test the framework harder.","The alpha sweep suggests a single global weight for BM25 versus embeddings is a simplification; a query-dependent alpha chosen by the router could push precision higher.","The consistent Korean-language drop points to the translation round-trip as a likely error source; evaluating retrieval directly on target-language embeddings, without translating to English first, would isolate that cause."],"forward_implications":["Archive users can ask natural-language questions and receive answer text that explicitly names the relevant file IDs, not just a ranked list.","Tuning the hybrid retriever's alpha to around 0.8 yields the best precision-F1 balance, so the BM25-to-vector weighting is a practical lever for archive search quality.","The system works across media types by converting images, audio, and video into AI-generated textual descriptions before indexing.","Non-English archives need language-specific handling: every tested model scored worse on Korean than on English, so translation alone does not fully close the gap.","The translator and query router are load-bearing: removing the router drops the F1 score by 38.73 percentage points and the hit rate by 36.36 percentage points."],"supporting_citations":[{"why":"Supplies the BGE-M3 embedding model used to create vector representations of file content and queries.","marker":"[1]"},{"why":"Provides the RAG survey that frames the retriever-generator architecture.","marker":"[2]"},{"why":"Motivates the reranking post-processor for long-document re-ranking.","marker":"[4]"},{"why":"Surveys retrieval-augmented generation for large language models, grounding the pipeline design.","marker":"[5]"},{"why":"Defines Mistral 7B, the LLM that achieves the reported 80.56% average precision.","marker":"[6]"},{"why":"Supports the long-context reordering technique that refocuses the model on key information.","marker":"[9]"},{"why":"Shows that hybrid sparse-dense retrieval improves scientific document retrieval, justifying the hybrid retriever.","marker":"[10]"},{"why":"Inspires the in-context retrieval-augmented generation approach used for response synthesis.","marker":"[13]"},{"why":"Supports hybrid query-based retrievers with semantic search for improved RAG accuracy.","marker":"[14]"},{"why":"Defines Llama 2, the weakest baseline LLM in the comparison.","marker":"[17]"}],"fun_headline_variants":["LLM-RAG archive search hits 80.56% precision","Mistral 7B RAG archive search: 80.56% precision","Hybrid retriever + query router: archive search at 80.56%","Smart archive search: Mistral 7B achieves 80.56% precision","RAG pipeline for archives: Mistral 7B precision 80.56%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation stands on treating the file IDs the LLM writes into its generated answer as the retrieval result; if the model hallucinates IDs or the synthetic relevance labels are inaccurate, the reported precision, recall, F1, and hit rate no longer describe actual retrieval quality.","fun_headline_variants_meta":{"raw":{"variants":["LLM-RAG archive search hits 80.56% precision","Mistral 7B RAG archive search: 80.56% precision","Hybrid retriever + query router: archive search at 80.56%","Smart archive search: Mistral 7B achieves 80.56% precision","RAG pipeline for archives: Mistral 7B precision 80.56%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1318,"prompt_tokens":815,"completion_tokens":503,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":431,"completion_tokens_details":{"reasoning_tokens":397}},"tokens_in":431,"tokens_out":503,"duration_ms":4288,"temperature":1.0,"reasoning_tokens":397,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:48:48.153525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators re-run the 110 queries and check whether every file ID the LLM cites exists in the archive and matches the requested file type and topic; if a substantial share of cited IDs are hallucinated or irrelevant, the 80.56% precision claim collapses. A second check would repeat the pipeline on a real archival corpus with real user queries and compare against human relevance judgments.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BGE-M3 embedding model used to create vector representations of file content and queries."}],"review_version":1}