{"id":"276e4bbf-c05d-4164-935d-bce4e74d62e1","arxiv_id":"2412.16412","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A RAG-based chatbot for FHWA bridge technology pages reports 95% accuracy on 15 self-defined similarity-threshold test questions, with no released artifacts or external validation.","lead":"Researchers built InfoTech Assistant, a chatbot that answers bridge inspection questions by retrieving text and images from the FHWA InfoTechnology website with a large language model and retrieval-augmented generation. The pilot reports 95% accuracy, but the evaluation uses a similarity threshold against the source text itself, with no code, data, or human evaluation released.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 95% accuracy claim rests on a circular metric: cosine similarity to the very source passage the RAG pipeline retrieves and feeds to the LLM, so high scores are expected regardless of factual correctness.","rationale":"The reader's weakest assumption is exactly the circularity of the cosine-similarity metric: expected responses are the scraped source passages that the RAG system retrieves and feeds to the LLM. My reading of Sections III–IV confirms this: the JSON database stores the scraped text, the 'Bot response' is directly generated from that scraped data (IV-C), and the evaluation vectorizes 'expected and actual responses' (IV-D-3) without stating the reference source, but the only candidate references are those same passages. The paper's own conclusion admits the metric's unreliability, which further weakens the claim. The internal inconsistency in Table VI (similarity 0.91 paired with accuracy 0.93 under a 0.85 threshold) independently signals that the reported accuracy numbers are not derived from the stated procedure. I therefore agree with the reader's REJECT verdict: the central empirical claim is not supported by the evidence as presented. A concrete test with independently authored gold answers would settle whether the metric is salvageable, but until such a test is performed, the paper's headline accuracy should not be accepted as a measure of factual correctness.","tokens_in":10977,"tokens_out":2715,"duration_ms":26036,"concrete_test":"Replace the reference in Equation (3) with independently authored gold answers: have a bridge-inspection engineer (not involved in the scraping) write one-sentence correct answers to the 15 test questions, then compute cosine similarity between the assistant's response and these gold answers using the same SentenceTransformer model and 0.85 threshold. If human-judged accuracy on the same 15 queries (by two blinded experts rating correctness) is materially below the reported 95%/93%, the original metric overstates true answer quality. This isolates whether the evaluation measures factual correctness or mere fidelity to the retrieved source.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of roughly 95% accuracy (Section IV-D-3, Eq. 3) is supported only by a self-referential evaluation. The 'expected' responses are the scraped FHWA source passages stored in the JSON database; the system retrieves those exact passages and the LLM is prompted to answer from that context. The 'actual' response is therefore a paraphrase or summary of the reference text. Cosine similarity between a generated answer and the source passage it was built from measures lexical/semantic overlap, not whether the answer is correct for the user's question. Any RAG system with a competent LLM would score highly on this metric even if it occasionally hallucinated details, omitted key constraints, or answered a slightly different question. The paper even acknowledges in Section V that similarity 'may not be the most reliable metric' because the Llama model produced correct but more detailed answers with lower similarity scores — an admission that the threshold does not track correctness. Additionally, the per-question accuracy values in Table VI are internally inconsistent with the stated threshold: for a single test case, a similarity above 0.85 would yield 100% accuracy, not the 0.89–0.99 values shown, suggesting either multiple unstated trials or an informal mapping. The metric is never validated against human judgment, an independent QA benchmark, or any ground-truth set beyond the scraped text itself. Thus the 95%/93% accuracy figures do not establish that the assistant reliably answers domain queries correctly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a pilot multimodal conversational agent, 'InfoTech Assistant,' for the FHWA InfoTechnology web portal. The system scrapes 41 bridge-technology pages, stores text and images in JSON, and uses a Flask back end with either Llama 3.1 8B or Mistral-7B-Instruct-v0.2 together with sentence-embedding-based retrieval (all-mpnet-base-v2) to answer domain-specific questions. The central reported result is that the assistant achieves about 95% (Llama 3.1) and 93% (Mistral 7B) accuracy on 15 bridge-technology queries, where correctness is defined by a cosine similarity of at least 0.85 between the generated response and an 'expected' response (Section IV-D-3, Eq. 3). The paper also reports latency ranges, sample interactions, and a qualitative discussion of retrieval limitations.","tokens_in":11241,"tokens_out":2795,"duration_ms":27128,"significance":"If the reported accuracy were valid, the work would be a modest but useful demonstration of a RAG-based domain chatbot for infrastructure professionals. The system integration (web scraping, JSON storage, embedding-based retrieval, local LLM serving) is clearly described and the appendix provides concrete sample data. However, the central quantitative claim is not supported: the evaluation metric is self-referential, comparing generated answers against the very source passages the RAG pipeline retrieves and supplies to the LLM. There is no human evaluation, no independent ground truth, no held-out test set, and no error analysis beyond a brief acknowledgment in Section V that cosine similarity 'may not be the most reliable metric.' The paper's own stated limitation, together with the internal inconsistency of Table VI, makes the headline accuracy figures unsubstantiated. The contribution is therefore reduced to a system-description pilot, not a validated evaluation of response quality.","major_comments":[{"comment":"The accuracy definition is circular. 'Expected' responses are evidently the scraped FHWA source passages stored in the JSON database (as shown in Tables III and IV), and the RAG system retrieves those exact passages before the LLM summarizes or paraphrases them into the 'actual' response. Cosine similarity between the generated answer and the source text it was constructed from measures lexical-semantic overlap, not factual correctness for the user's question. No external labels, human ratings, or independent benchmark are used. The paper itself concedes in Section V that in some cases Llama produced 'correct and detailed answers' with lower similarity, which directly undercuts the claim that the 0.85 threshold tracks correctness. The 95%/93% accuracy figures therefore do not establish the assistant's reliability.","section":"Section IV-D-3, Eq. (3)"},{"comment":"The per-question accuracy values are internally inconsistent with the stated definition. Equation 3 defines accuracy as the percentage of correct responses among total test cases, and Section IV-D-3 states that a response is correct if its cosine similarity is 0.85 or higher. For a single test case this yields exactly 100% or 0%, yet Table VI reports accuracies such as 0.91, 0.93, and 0.99. If there were multiple repeated trials per question, the paper does not say so, and the accuracy values are suspiciously close to the corresponding similarity scores, suggesting either an informal mapping or unstated aggregation. This inconsistency makes the quantitative evaluation impossible to interpret and requires correction or explicit clarification of the testing protocol.","section":"Table VI and Section IV-D-3"},{"comment":"The evaluation lacks statistical rigor for the claims made. Only 15 questions are used, with no description of how they were selected, whether they cover the full set of 41 technologies, or whether they are a held-out set. Despite the text in Section IV claiming testing with 'technical and non-technical users' and 'multiple testing rounds,' no user study protocol, participant numbers, or satisfaction measurements are reported. Latency ranges (15-20 seconds for Llama, 10-22 for Mistral) are given without error bars or condition details beyond the single hardware configuration in Table V. The absence of these details, combined with the metric problems above, means the paper's evaluative claims are not reproducible from the information provided.","section":"Section IV-D and Section IV-D-2"}],"minor_comments":[{"comment":"There is a typo in the introduction: 'to to enhance information retrieval' should read 'to enhance information retrieval.'","section":"Section I"},{"comment":"Equation (1) as printed is incomplete: the denominator is written as 'P exp(zj/T )' instead of a sum over j of exp(zj/T), making the softmax formula incorrect as typeset.","section":"Section III-E-3, Eq. (1)"},{"comment":"The text states that cosine similarity values 'calculated using Equation 3,' but Equation 3 is the accuracy formula; the cosine similarity is defined in Equation 2. The cross-reference should be corrected.","section":"Section IV-D-3"},{"comment":"The discussion of all-mpnet-base-v2 cites reference [5] (Xiao, 2022), but that reference appears to describe a different QA chatbot model, not the Microsoft sentence-transformer model. The citation should be replaced with the actual all-mpnet-base-v2 source.","section":"Section III-C and Reference [5]"},{"comment":"The acknowledgment refers to 'the LLaMA model' and thanks Meta and Mistral; for consistency with the rest of the paper, the Llama 3.1 model name should be used, and formatting inconsistencies such as 'FHW A' should be cleaned up.","section":"Acknowledgment"},{"comment":"The table would benefit from a column explaining the number of trials or repetitions per question, since the reported accuracy values appear to be fractional rather than binary; without this information the table is misleading.","section":"Table VI"}],"recommendation":"reject","confidential_remarks":"The central evaluation is not merely weak but self-referential, and Table VI contains an internal inconsistency that cannot be fixed by rephrasing. Rejecting is appropriate because the headline accuracy claim would need to be either retracted or replaced with a completely different evaluation protocol (human judgment, a constructed QA benchmark, or a held-out factoid set) before the paper's main quantitative contribution becomes valid. The system description itself may be salvageable as a short demonstration paper, but not with the current evaluation presented as a rigorous comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a standard RAG chatbot applied to one FHWA web portal, and the headline 95% accuracy figure rests on a metric that doesn't test correctness. What's genuinely here is a clear system description: they scraped 41 bridge-technology pages, stored the text and images in JSON, and built a Flask app with Llama 3.1 and Mistral 7B behind it. The appendix tables on Hammer Sounding and Magnetic Particle Testing give a concrete picture of the data. The latency measurements and model comparison are honest, and the authors acknowledge in Section V that the similarity metric has limitations.\n\nThe soft spots are in the evaluation. Equation 3 defines accuracy as the fraction of responses whose cosine similarity to the expected answer exceeds 0.85. The expected answer is the scraped source passage, and the RAG system retrieves that exact passage and feeds it to the LLM as context. So the similarity score measures how close the model's output is to its own input text, not whether the answer is factually correct for the user's query. The paper's own Section V admission—that Llama gave correct but more detailed answers with lower similarity—shows the metric does not track correctness.\n\nTable VI is internally inconsistent with the stated method. Every reported similarity score is above the 0.85 threshold, so per the paper's own definition every question should be 100% accurate. Instead the table shows per-question accuracy values from 0.89 to 0.99. That doesn't match the binary correct/incorrect calculation in Equation 3. Either there were multiple unstated trials per question, or the accuracy column is doing something else; neither possibility is explained.\n\nThere's also no human evaluation, no held-out set, no error bars, and no released code or data. Fifteen questions is too small for a meaningful percentage claim. The citation list is sloppy: the sentence-transformer model is attributed to the wrong paper, and several references are off-topic.\n\nNet: this is a pilot demo, not a research contribution. The system could be a useful internal tool for bridge inspectors, and the paper is an honest write-up of a standard pipeline. But the central accuracy claim is not supported, and the evaluation is self-referential. I would not send this to serious peer review in its current form. If the authors redo the evaluation with human judgments or an independent QA benchmark, it might become a small, publishable systems paper. As it stands, a desk reject is the right call.","headline":"A standard RAG chatbot for one FHWA portal, with a headline accuracy claim that rests on a circular metric and an internally inconsistent evaluation table.","tokens_in":11799,"tokens_out":4183,"would_cite":false,"duration_ms":35249,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAG chatbot answers bridge-tech lookup queries at ~95% accuracy.","keywords":["retrieval-augmented generation","bridge inspection","multimodal chatbot","Llama 3.1","Mistral 7B","cosine similarity","web scraping","infrastructure technology"],"falsifier":"Have a bridge-inspection expert independently score the 15 test answers as correct, incorrect, or partial, and compare those labels with the 0.85 cosine threshold; if many answers above the cutoff are judged wrong, or many below it are judged right, the accuracy metric is not measuring answer correctness.","tokens_in":10779,"feed_emoji":"🏗️","tokens_out":6784,"duration_ms":49903,"temperature":0.7,"pith_summary":"The InfoTech Assistant is a pilot chatbot that answers lookup questions about bridge inspection and assessment technologies by retrieving scraped web-portal pages and letting an LLM generate the reply. The paper's central claim is that this retrieval-augmented system answers domain-specific queries correctly about 95 percent of the time with Llama 3.1 and about 93 percent with Mistral 7B, where 'correct' means the vectorized answer scores at least 0.85 cosine similarity to the expected response. The system also returns relevant images from the portal alongside text, and pairs a source-based 'bot' answer with a shorter LLM summary. If the claim holds, it shows that a modest, locally hosted model plus a small structured database can serve as a dependable reference tool for infrastructure professionals.","feed_headline":"RAG chatbot hits ~95% accuracy on bridge-tech queries","feed_subtitle":"Llama 3.1 answers portal lookup questions from scraped source pages, scored by cosine similarity.","key_machinery":"The load-bearing mechanism is a retrieval-augmented generation (RAG) loop: a JSON database of scraped pages, an all-mpnet-base-v2 embedding model that retrieves the closest passage, and an LLM that generates the answer from that passage. The evaluation identity is Equation 3: accuracy is the share of test cases whose cosine similarity between the expected and actual response embeddings is at least 0.85. The same embedding model does double duty, both finding relevant context and scoring whether the answer matches.","core_discovery":"The paper reports that a RAG chatbot built on 41 scraped bridge-technology pages can handle conversational queries such as 'What is Electrical Resistivity?' and 'What are benefits of Hammer Sounding?' with high similarity to expected answers. Over 15 test questions, Llama 3.1 8B reaches an overall accuracy of 95 percent and Mistral 7B reaches 93 percent, with latency between roughly 10 and 22 seconds on a local workstation. The authors frame the contribution as a domain-specific, multimodal conversational agent: retrieval is grounded in the portal's own text, and images are pulled from the same source and displayed with the reply. They also note the dual-response format, where the bot response reproduces the source passage and the LLM response summarizes it, and they acknowledge that cosine similarity can underrate a factually correct answer that adds explanatory content.","pith_inferences":["In the editor's view, the 95 percent figure is best read as retrieval-and-paraphrase fidelity, not independent factual accuracy, because the expected response is the same scraped passage the retriever feeds the model.","A testable extension: run the same pipeline on a second technical portal and check whether the 0.85 cosine threshold keeps accuracy above 90 percent without retuning.","The evaluation is likely sensitive to question phrasing; a correct answer expressed differently from the source text could score below 0.85 even though a human would accept it."],"forward_implications":["A locally hosted 8B parameter model, running without cloud APIs, can reach roughly 95 percent accuracy on a narrow technical vocabulary drawn from the portal.","Llama 3.1 8B outperforms Mistral 7B on similarity-defined accuracy, while Mistral's latency is comparable or lower, so model choice becomes a speed-versus-accuracy trade-off.","The dual-response design means a professional can read the verbatim source passage for precision and the LLM summary for a quick overview.","The same scraping-to-JSON-to-RAG pipeline can be extended to the pavements, tunnels, and utilities sections already included in the dataset."],"supporting_citations":[{"why":"supplies the scraped source text and images for 41 bridge technologies.","marker":"[4]"},{"why":"provides the retrieval-augmented generation approach the system builds on.","marker":"[8]"},{"why":"supports the hybrid RAG technique used to ground LLM query responses.","marker":"[24]"},{"why":"defines the all-mpnet-base-v2 embedding model used for retrieval and similarity scoring.","marker":"[33]"},{"why":"is cited as the Sentence-Transformer vectorizer for expected and actual responses.","marker":"[5]"},{"why":"underlies the Llama 3.1 8B model that generates the assistant's answers.","marker":"[28]"},{"why":"underlies the Mistral 7B comparison model.","marker":"[26]"},{"why":"supports cosine similarity as the measure of response relevance.","marker":"[7]"}],"fun_headline_variants":["Bridge-tech chatbot answers with 95% accuracy","RAG chatbot nails 95% on bridge queries","Multimodal RAG bot aces bridge tech Q&A","Llama-based RAG chatbot hits 95% on bridge info","95% accuracy: RAG chatbot for bridge tech queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy rests on the assumption that a cosine similarity of 0.85 or higher, measured against a response drawn from the very passage the system retrieves, is a valid test of whether an answer is correct.","fun_headline_variants_meta":{"raw":{"variants":["Bridge-tech chatbot answers with 95% accuracy","RAG chatbot nails 95% on bridge queries","Multimodal RAG bot aces bridge tech Q&A","Llama-based RAG chatbot hits 95% on bridge info","95% accuracy: RAG chatbot for bridge tech queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1369,"prompt_tokens":895,"completion_tokens":474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":511,"tokens_out":474,"duration_ms":4221,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:36:01.209867+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a bridge-inspection expert independently score the 15 test answers as correct, incorrect, or partial, and compare those labels with the 0.85 cosine threshold; if many answers above the cutoff are judged wrong, or many below it are judged right, the accuracy metric is not measuring answer correctness.","supporting_citations":[{"cited_title":"Bridge FHW A InfoTechnology,","cited_arxiv_id":null,"evidence_quote":"supplies the scraped source text and images for 41 bridge technologies."},{"cited_title":"GitHub - alfredodeza/learn-retrieval-augmented- generation: Examples and demos on how to use Retrieval Augmented Generation with Large Language Models","cited_arxiv_id":null,"evidence_quote":"provides the retrieval-augmented generation approach the system builds on."}],"review_version":1}