{"id":"0b0ac5ca-f13a-4d28-81dd-73e8fa184f11","arxiv_id":"2506.23026","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A RAG-based tutoring and support chatbot with hybrid search and an instructor feedback loop is described, but no quantitative evaluation of its accuracy is provided.","lead":"This paper describes MARK, a chatbot that answers student questions by pulling answers from a course knowledge base instead of relying on the model's memory alone. It is a readable example of a widely used approach, retrieval-augmented generation, applied to education and customer support, but it offers only anecdotes instead of measurements.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central reliability claims rest on anecdotal deployment stories; no retrieval, faithfulness, or hallucination metrics are reported in Section 5.","rationale":"The reader's REJECT verdict is based on the absence of any empirical validation of the accuracy and hallucination-reduction claims. My stress-test confirms this is the load-bearing weakness: the paper's own text, and in particular Section 5, contains no measurements whatsoever of the system's reliability. The architecture sections (2-4) are standard RAG descriptions, and the prompt in Section 4.3 does not even enforce strict grounding, which makes the claimed reduction of hallucination an assumption rather than a demonstrated property. Because the claimed contribution is a deployed system whose stated value is reliability, the missing evaluation is not a minor omission but the central evidence gap. I did not find an internal inconsistency in the RAG pipeline itself, and I do not treat the system's use of standard components as a flaw; the issue is that the paper draws a strong performance conclusion without measuring it. The concrete test I propose (public deployment logs plus retrieval/faithfulness/hallucination metrics with baselines) would either substantiate or refute the central claim. Since the reader already rejected on this ground and no new corrective factor emerged, the verdict remains REJECT. I agree with the reader's weakest_assumption that grounding is assumed rather than demonstrated; I would only add that the tunable external-knowledge parameter and the feedback-loop description reinforce that the system's behavior is not characterized at all.","tokens_in":12546,"tokens_out":1653,"duration_ms":15429,"concrete_test":"To settle the concern, the authors should release the Section 5.1 deployment logs (or a sanitized sample) and report standard RAG evaluation numbers: retrieval precision/recall at top-k, answer faithfulness (e.g., percent of claims attributable to retrieved chunks), and a hallucination rate defined with a rubric, plus a baseline comparison (e.g., the same LLM without RAG and BM25-only retrieval). If faithfulness is below, say, 90% on a held-out set of the logged queries, or if there is no measurable difference from the no-RAG baseline, the central reliability claim fails; if such numbers meet the stated rubric, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The conclusion and abstract claim that MARK 'is able to generate more accurate and trustworthy responses, reducing the risk of hallucination' and that the classroom deployment 'successfully addressed a broad range of student queries.' The only evidence in Section 5 is a narrative description (Section 5.1) and four illustrative technical-support dialogues (Section 5.2). No evaluation is reported: no query counts, no accuracy or answer-faithfulness measurements, no retrieval precision/recall, no hallucination rate, no comparison against a baseline (e.g., LLM without RAG, BM25-only, or the ChemTAsk system cited in [34]). Moreover, the direct textual grounding mechanism is weak: the prompt in Section 4.3 says only 'politely decline if a question is outside your knowledge' and 'be concise'; it does not instruct the model to answer exclusively from retrieved chunks or to abstain when retrieved content is insufficient, and Section 3.2 even exposes a tunable parameter (0-100) controlling how much external knowledge the LLM may use, without stating the deployed value. The instructor-correction feedback loop in Section 4.4 is described as updating indexes, but no evidence is given that corrections are retrieved and applied correctly. The claim that the system 'reduces hallucination' is therefore not derivable from the reported components; it is an unsupported assertion, and Section 5's own admission of limited adoption ('Overall adoption during the 2023 pilot phase was limited') further undercuts the generalization that it 'successfully addressed a broad range of student queries.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes MARK, a retrieval-augmented generation (RAG) system that combines BM25 sparse retrieval with FAISS-based dense retrieval, a re-ranking stage, an instructor-feedback loop, and a web frontend. The authors report a pilot deployment in a Fluid Mechanics course and a technical-support use case, and they conclude that MARK 'is able to generate more accurate and trustworthy responses, reducing the risk of hallucination.' Sections 2 and 4 present standard retrieval and backend techniques; Sections 3 and 5 describe the interfaces and anecdotal deployments.","tokens_in":12849,"tokens_out":4578,"duration_ms":45686,"significance":"The paper's potential contribution is a deployable RAG system for education with instructor control and feedback. The system is publicly accessible, and the retrieval implementation follows well-established methods. However, the paper offers no quantitative evaluation of retrieval effectiveness, answer faithfulness, or hallucination rates, and the only evidence in Section 5 is narrative and illustrative. As a result, the central claims of accuracy, reliability, and reduced hallucination are not demonstrated; the significance of the work to the IR community is therefore not established.","major_comments":[{"comment":"Section 5 contains no quantitative evaluation. The abstract and conclusion claim that MARK 'successfully addressed a broad range of student queries' and 'is able to generate more accurate and trustworthy responses, reducing the risk of hallucination,' yet Section 5.1 offers only a narrative description and Section 5.2 presents four illustrative dialogues. There are no query counts, accuracy or answer-faithfulness metrics, retrieval precision/recall numbers, hallucination-rate measurements, or comparisons against a no-RAG baseline or alternative retrievers. These claims therefore have no evidential support.","section":"Section 5 (Results)"},{"comment":"The prompt in Section 4.3 does not instruct the LLM to answer exclusively from retrieved content; it says only to 'politely decline if a question is outside your knowledge' and to 'be concise.' Moreover, Section 3.2 exposes a tunable parameter controlling how much external knowledge the LLM may use (0-100), but the deployed value is never stated. Thus the mechanism claimed to reduce hallucination is not guaranteed by the design, and no faithfulness measurement is reported to verify that responses are grounded in the retrieved passages.","section":"Sections 4.3 and 3.2"},{"comment":"The Fluid Mechanics corpus was seeded with 'curated answers to student questions by instructors.' As a result, some successful responses may have been pre-written text retrieved from the corpus rather than answers generated by the LLM from retrieved evidence. The paper does not distinguish these cases, so the classroom results do not directly support the claim that RAG improves LLM generation. The manuscript's own statement that 'Overall adoption during the 2023 pilot phase was limited' further weakens the claim of successful deployment.","section":"Section 5.1"}],"minor_comments":[{"comment":"The BM25 IDF formula in Eq. (6) appears to include an extra '+1' term inside the logarithm; please check whether this is intentional or a typesetting error relative to the standard formulation.","section":"Eq. (6)"},{"comment":"There are several typos: 'weigh' for 'weight', 'acorss' for 'across', 'docment' for 'document', 'conduced' for 'conducted', and 'BERTA' for 'BERT' in the example list of cross-encoder models in Eq. (12).","section":"Throughout"},{"comment":"The paper does not report the values of hyperparameters such as k1, b, hybrid fusion weights, or top-k; please state them explicitly or clarify that system defaults were used.","section":"Sections 2 and 4.2"}],"recommendation":"reject","confidential_remarks":"The manuscript is best characterized as a system/application description. The lack of any formal evaluation is a fundamental problem for a research paper; however, the public deployment and the described feedback loop might be of interest as a short demo or industry-practice paper. The retrieval mathematics in Sections 2 and 4 are standard and correctly stated, so the rejection is based on the absence of evidence for the central claims rather than on technical errors in the retrieval formulation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: the architecture is standard and the paper knows it; the new contribution is essentially a deployed system with an instructor feedback loop. But there is no evaluation anywhere in Section 5, so the central claim about reducing hallucination is unsupported.\n\nWhat the paper does well: it writes clearly about BM25, dense retrieval, FAISS, cross-encoder reranking, and the feedback loop. The math is correct and standard. It also cites prior RAG tutoring systems (ChemTAsk, AI-U, RAGMan), so it does not pretend to invent RAG. The deployment story is honest about limited adoption and about the system giving complete solutions instead of step-by-step guidance.\n\nThe soft spots: Section 5 is a narrative with four illustrative dialogues. No query counts, no accuracy or faithfulness metrics, no hallucination rate, no comparison against a non-RAG LLM or a BM25-only baseline. The prompt in Section 4.3 says only \"politely decline if a question is outside your knowledge\" and \"be concise\"; it does not tell the model to answer exclusively from retrieved chunks. And Section 3.2 exposes a tunable parameter (0-100) controlling how much external knowledge the LLM may use, without stating the deployed value. That makes the grounding claim hard to pin down. Also, the Fluid Mechanics corpus was seeded with instructor-written answers to student questions, so some successes may be retrieved rather than generated. That is not circular in a formal sense, but it weakens the generalization from the anecdote.\n\nMinor point: the conclusion says \"By integrating carefully curated knowledge with LLMs, MARK is able to generate more accurate and trustworthy responses\" — that is a claim the paper does not test. The paper itself admits adoption was limited, which undercuts the abstract's \"successfully addressed a broad range of student queries.\"\n\nWho is this for: someone wanting a worked example of a hybrid RAG chatbot with a feedback loop might get a bit of value. But as a research paper, it is not ready. I would not send it to peer review in its current form; I would tell the authors to add even basic usage statistics and a small faithfulness evaluation. If they did that, it could be a serviceable system paper.\n\nMy recommendation: desk reject with an invitation to resubmit with evaluation.","headline":"A clear system description of a standard hybrid RAG chatbot, but the central reliability claims rest entirely on anecdote and are not measured.","tokens_in":13364,"tokens_out":1766,"would_cite":false,"duration_ms":17913,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces MARK, a retrieval-augmented chatbot that grounds answers in a curated course knowledge base, and claims this design reduces hallucination risk and can substitute for office hours and routine technical support.","keywords":["retrieval-augmented generation","RAG","hallucination reduction","hybrid search","BM25","dense retrieval","educational chatbot","feedback loop"],"falsifier":"Pose a set of course questions whose correct answers are deliberately absent from the knowledge base and a matched set whose answers are present, then check whether the system frequently answers the absent set with confident but wrong content instead of politely declining; any substantial rate of unsupported answers would falsify the groundedness claim.","tokens_in":12325,"feed_emoji":"🎓","tokens_out":5605,"duration_ms":59721,"temperature":0.7,"pith_summary":"MARK is a retrieval-augmented chatbot that answers each student or customer question by hunting through a curated knowledge base with both exact keyword matching and semantic vector search, merging the hits, and handing the top passages to a large language model as the basis for a concise answer. The paper claims this grounded design produces more accurate and trustworthy responses and reduces the hallucination risk of answering from memory alone. A feedback loop lets instructors rewrite poor answers, and the corrections are indexed back into the corpus, so the system improves with use. Deployed in a Fluid Mechanics course as a stand-in for office hours and in a customer-support setting, MARK reportedly handled a broad range of routine queries. A sympathetic reader would take the contribution as a recipe: RAG plus instructor curation and feedback can give students a reliable, scalable 24/7 assistant.","feed_headline":"Course-grounded AI tutor cuts hallucination risk","feed_subtitle":"MARK answers student questions from a curated knowledge base and improves as instructors correct it.","key_machinery":"The load-bearing mechanism is a hybrid retrieval corpus with a feedback loop. Queries run in parallel through BM25, which matches exact terms, and a dense vector search over 512-dimensional Jina embeddings indexed with approximate-nearest-neighbor search; the top results are merged, re-ranked, and the top k passages become context for the language model. The prompt asks the model to be a patient teaching assistant, refer to prior conversation, politely decline out-of-knowledge questions, and be concise. Instructor corrections entered through a monitoring panel are appended to the raw document store, and both the sparse and dense indexes are regenerated, making the knowledge base the system's memory rather than the model's parameters.","core_discovery":"The paper's central claim is that a retrieval-augmented generator grounded in a curated, instructor-maintained knowledge base can deliver accurate and trustworthy responses, sufficiently so that it can stand in for office hours and handle routine technical support. The mechanism is MARK: BM25 keyword retrieval and dense embedding retrieval run in parallel, the results are merged and re-ranked, the top passages go to a language model under a short prompt, and instructor corrections are written back into the corpus. The paper reports deployment in a Fluid Mechanics course and in a customer-support setting, where the system successfully addressed a broad range of student queries and answered routine customer questions. The intended consequence is reduced hallucination relative to a bare language model, because outputs are anchored to curated passages rather than the model's parametric memory.","pith_inferences":["A testable implication left implicit: if the feedback loop is the main driver of reliability, then a version of MARK without instructor corrections should show measurably more hallucination on the same corpus; the paper does not isolate this effect.","The same design would plausibly transfer to other long-tail knowledge domains, such as internal policy help or clinical triage, where the corpus can be kept curated and current; that extension is not claimed by the paper.","The paper's own limitation note points to multimodal input as a natural next step, since text-only retrieval cannot catch student errors in equations or diagrams.","The 384-token chunk length and the practice of including section headings are testable design decisions; ablating them would reveal how much of the reliability comes from retrieval parameters rather than from the language model."],"forward_implications":["An instructor can deploy MARK as a 24/7 substitute for routine office hours, absorbing logistical and homework queries without additional teaching-staff time.","Every corrected answer becomes part of the retrieval corpus, so the system is expected to improve with use.","Hybrid BM25-plus-dense retrieval covers both exact-term queries and paraphrased semantic queries, making the assistant more robust to different question styles.","Swapping the curated corpus lets the same system serve technical support, including account lookups and troubleshooting, with natural-language answers.","MARK is not a full replacement for human support, because some students and customers still prefer or request interaction with a real person."],"supporting_citations":[{"why":"Defines retrieval-augmented generation as the architecture for knowledge-intensive NLP tasks and motivates the paper's core design.","marker":"[28]"},{"why":"Surveys hallucination in natural language generation, establishing the failure mode MARK aims to reduce.","marker":"[20]"},{"why":"Provides evidence that retrieval augmentation reduces hallucination in conversation, the central premise behind MARK.","marker":"[27]"},{"why":"Supplies the BM25 probabilistic retrieval formula used as the sparse component of hybrid search.","marker":"[41]"},{"why":"Provides the dense passage retrieval approach that underpins the semantic side of MARK's hybrid retrieval.","marker":"[43]"},{"why":"Describes the approximate-nearest-neighbor search library used to make dense retrieval fast and scalable.","marker":"[44]"},{"why":"Supplies the Jina embeddings model chosen to encode text chunks into 512-dimensional vectors.","marker":"[50]"},{"why":"Reports a prior RAG-enabled course assistant that matched human teaching assistants, a deployment pattern MARK extends.","marker":"[34]"},{"why":"Describes an earlier instructor-aligned RAG system in a graduate engineering course, giving context for MARK's classroom use.","marker":"[35]"},{"why":"Documents a RAG-based programming tutor with high in-scope accuracy, supporting the claim that RAG tutors work in classroom settings.","marker":"[36]"}],"fun_headline_variants":["RAG tutor anchors answers to curated knowledge","Hybrid search RAG serves accurate student answers","AI tutor improves via instructor feedback loop","Curated RAG knowledge base powers reliable tutor","MARK stands in for office hours with grounded QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire reliability claim rests on two premises that the paper does not measure: that the language model, given the retrieved chunks and the MARK prompt, will stick to those chunks rather than its own memorized knowledge, and that the curated corpus contains accurate and complete answers; if either premise fails, the claimed reduction in hallucination is not assured.","fun_headline_variants_meta":{"raw":{"variants":["RAG tutor anchors answers to curated knowledge","Hybrid search RAG serves accurate student answers","AI tutor improves via instructor feedback loop","Curated RAG knowledge base powers reliable tutor","MARK stands in for office hours with grounded QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000882,"raw_usage":{"total_tokens":3773,"prompt_tokens":870,"completion_tokens":2903,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":2835}},"tokens_in":486,"tokens_out":2903,"duration_ms":20758,"temperature":1.0,"reasoning_tokens":2835,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:52:05.070936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pose a set of course questions whose correct answers are deliberately absent from the knowledge base and a matched set whose answers are present, then check whether the system frequently answers the absent set with confident but wrong content instead of politely declining; any substantial rate of unsupported answers would falsify the groundedness claim.","supporting_citations":[{"cited_title":"Lewis, E","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation as the architecture for knowledge-intensive NLP tasks and motivates the paper's core design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys hallucination in natural language generation, establishing the failure mode MARK aims to reduce."},{"cited_title":"Robertson and H","cited_arxiv_id":null,"evidence_quote":"Supplies the BM25 probabilistic retrieval formula used as the sparse component of hybrid search."},{"cited_title":"Karpukhin, B","cited_arxiv_id":null,"evidence_quote":"Provides the dense passage retrieval approach that underpins the semantic side of MARK's hybrid retrieval."},{"cited_title":"Johnson, M","cited_arxiv_id":null,"evidence_quote":"Describes the approximate-nearest-neighbor search library used to make dense retrieval fast and scalable."},{"cited_title":"Jina embeddings 2: 8192-token general-purpose text embeddings for long documents, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the Jina embeddings model chosen to encode text chunks into 512-dimensional vectors."},{"cited_title":"Large Language Models for Education: ChemTAsk -- An Open-Source Paradigm for Automated Q&A in the Graduate Classroom","cited_arxiv_id":"2502.00016","evidence_quote":"Reports a prior RAG-enabled course assistant that matched human teaching assistants, a deployment pattern MARK extends."},{"cited_title":"Faghih Shojaei, R","cited_arxiv_id":null,"evidence_quote":"Describes an earlier instructor-aligned RAG system in a graduate engineering course, giving context for MARK's classroom use."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents a RAG-based programming tutor with high in-scope accuracy, supporting the claim that RAG tutors work in classroom settings."}],"review_version":1}