{"id":"08270436-112a-4d6e-8871-cd51548253c4","arxiv_id":"2411.15491","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Combining Jieba keyword matching with retrieval-augmented generation and reranking produces a small score increase for TCM case diagnosis, but the score is undefined and the data are not released.","lead":"This paper describes an AI system that gathers thousands of Traditional Chinese Medicine case studies from websites and uses retrieval and prompting to help language models answer diagnosis questions about them. The authors report better scores when they add keyword matching, but they never define what the scores measure.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The only quantitative support for the hybrid-retrieval claim is Table 1, but the score is undefined and the 0.90-point difference has no error bars or significance test, so 'significantly enhanced' is not established.","rationale":"I read the paper in good faith as a systems/application description rather than a rigorous benchmarking study. The proposed pipeline (web-scraped TCM database, ERNIE-based cleaning, hybrid retrieval, and CoT prompting) is plausible, and the components are recognized techniques. However, the abstract's quantitative claim is 'significantly enhanced accuracy,' and the only numeric evidence is Table 1. That table is not self-contained: the metric, evaluation set, and comparison procedure are absent. The 0.90-point gain is reported without variance, so 'significant' cannot be checked. A second table uses a different score scale (24–27), reinforcing that the metric is uncontrolled. Independent evidence is thin: no code, no data release, and no formal verification. The reference list contains unrelated entries (e.g., refs 1 and 13), which further weakens confidence but is not the central technical flaw. The concrete check is to obtain the evaluation protocol and bootstrap the per-case difference; if the confidence interval includes zero, the central claim fails. This supports the reader's REJECT verdict, so no adjustment is needed.","tokens_in":8582,"tokens_out":4654,"duration_ms":40356,"concrete_test":"Obtain or reconstruct the evaluation protocol behind Table 1: the Score metric, the test set, and the per-case scoring function. Then run a paired bootstrap (10,000 resamples) on the per-case score differences between 'Naive RAG' and 'Naive RAG + Jieba' on the same fixed test set. If the 95% confidence interval for the difference contains 0, or if the per-case scores and evaluation script cannot be provided, the 'significantly enhanced' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract and Section 4.2 is that adding Jieba keyword matching to RAG 'significantly enhanced the accuracy of model outputs.' The sole evidence is Table 1, showing 36.15 for 'Naive RAG' and 37.05 for 'Naive RAG + Jieba.' This table does not define the 'Score' metric, the number of test cases, the question set, or the scoring rubric. The accompanying text also conflates the two rows with 'RAG' versus 'Jieba,' so it is not clear what the comparison is actually isolating. Without error bars, confidence intervals, or a significance test, a 0.90-point difference is not interpretable as 'significant'; it could be a single question or run-to-run noise. Additionally, Table 2 reports base-model scores in the 24–27 range, versus Table 1's 36–37 range, implying different tasks or different score scales; this makes the absolute values uninterpretable across the paper. Because the paper's headline contribution is precisely this improvement, the missing evaluation protocol is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the construction of a Traditional Chinese Medicine (TCM) clinical case database (over 5,000 cases scraped from public web sources) and a retrieval-augmented generation (RAG) pipeline for syndrome-differentiation tasks. The pipeline uses semantic vector retrieval (gte-Qwen2-1.5B-instruct), Jieba keyword matching, a reranker (gte-passage-ranking-multilingual-base), and DeepSeekV2 as the generator, with chain-of-thought style prompts producing structured JSON outputs. The central claim, stated in the abstract and Section 4.2, is that combining two-stage retrieval with Jieba keyword matching 'significantly enhanced the accuracy of model outputs.' The empirical support consists of two tables of single 'Score' values, but the metric, test set, scoring rubric, and experimental procedure are never defined.","tokens_in":8950,"tokens_out":4719,"duration_ms":41420,"significance":"If the claims were substantiated, the dataset and hybrid retrieval pipeline could be a useful engineering contribution for TCM natural language processing, an area with few openly available structured clinical corpora. The system sensibly combines established components (RAG, reranking, keyword matching), and the dataset-construction effort addresses a real need. However, the paper provides no valid quantitative evidence for its central claim: the reported scores are undefined, the comparison in Section 4.2 is mislabeled, and no error bars or significance tests are given. As submitted, the contribution is unverified; a proper evaluation on a defined benchmark would be needed before the results could be considered reliable.","major_comments":[{"comment":"The 'Score' metric in Tables 1 and 2 is never defined, and the test questions, dataset split, and scoring procedure are not described. Without these, the 0.90-point difference between 36.15 and 37.05 in Table 1 cannot be interpreted, and the abstract's claim of 'significantly enhanced accuracy' is unsupported. This is a load-bearing issue because the entire contribution rests on this comparison.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The prose in Section 4.2 compares 'Jieba' with 'overlap window,' but Table 1 labels the rows 'Naive RAG' and 'Naive RAG + Jieba.' This mismatch makes it impossible to know which system configurations are being compared and whether the reported gain is due to Jieba, RAG, or their combination. The text also treats 'RAG' and 'Jieba' as competing methods, while the table suggests the comparison is between RAG with and without Jieba.","section":"Section 4.2, text vs. Table 1"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere in Section 4. Given the unknown number of test cases and the small reported difference (0.90 points), the observed improvement could be run-to-run noise or the effect of a single test question. Additionally, Table 2 reports base-model scores in the 24–27 range while Table 1 reports 36–37, implying different tasks or different score scales; this prevents any cross-table interpretation of absolute values.","section":"Section 4 (general)"},{"comment":"The qualitative examples in Figure 4 and Section 4.1 purport to show that RAG improves diagnostic reasoning, but no quantitative measure supports this. The chain-of-thought discussion in Section 3.2 similarly presents only a single illustrative example. These claims are not backed by experimental evidence and cannot substitute for the missing evaluation.","section":"Sections 3.2 and 4.1"}],"minor_comments":[{"comment":"There are numerous typos and formatting errors, such as 'T able 1,' missing spaces between words (e.g., 'theBaidu_ERNIE_Speed_128KAPI'), and inconsistent spellings of model names. The paper needs thorough proofreading.","section":"Throughout"},{"comment":"Reference [13] is 'Gender and Precarity in Astronomy' and reference [1] is about line spacing in LaTeX; neither appears relevant to the claims they support. The authors should verify that all references are appropriate and correctly cited.","section":"References"},{"comment":"Section 3.1 gives a URL for the dataset (https://cccl-tcm.cacm.org.cn/article?lang=zh) but does not specify licensing, access conditions, or whether the processed dataset is actually downloadable. This should be clarified.","section":"Section 3.1"},{"comment":"The paper claims 'over 5,000 TCM clinical cases' but provides no statistics about the corpus (e.g., number of syndromes, distribution of case lengths, or sources), so the reader cannot assess the dataset's coverage or representativeness.","section":"Section 3.1"},{"comment":"The figure captions are minimal and sometimes do not explain the claimed workflow; for example, Figure 3's caption does not describe the OCR and text-cleaning steps mentioned in the text.","section":"Figures"}],"recommendation":"reject","confidential_remarks":"The manuscript reads more like a technical blog post than a peer-reviewed research paper. The undefined evaluation metric, the mislabeled comparison in Section 4.2, and the irrelevant references suggest that the work is not yet at the stage of a journal submission. The fit with a cs.CL journal is also questionable because the contribution is primarily a systems/engineering description without a rigorous validation. I would be open to reconsidering a revised version that includes a proper evaluation protocol, defined metrics, and a clear comparison of the proposed hybrid method against baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper is a system report, not a validated study. It describes a sensible TCM clinical-case pipeline—web scraping over 5,000 cases, ERNIE-based cleaning, gte-Qwen2 embeddings, Jieba keyword matching, gte reranking, DeepSeekV2 generation—but its central claim of 'significantly enhanced accuracy' rests on one table with an undefined score. That claim is not supported.\n\nWhat is genuinely new is the specific assembly: applying hybrid retrieval with keyword and semantic matching to TCM syndrome differentiation, and forcing the output into structured JSON. That could be a useful component for practitioners if the dataset and code were shared. The method section is readable and the figures help.\n\nNow the soft spots, and they are load-bearing. Table 1 gives 36.15 for 'Naive RAG' and 37.05 for 'Naive RAG + Jieba' without ever saying what Score measures, how many questions were used, what the scoring rubric is, or whether there is any variance. A 0.90-point difference with no error bars cannot be called significant. The prose even mislabels the comparison, discussing 'Jieba' vs 'RAG' when the table labels the rows differently. Table 2 reports scores in the 24–27 range, which implies a different task or scale, so the absolute numbers do not reconcile. There is no external benchmark, no released dataset, and no code. The reference list is also sloppy—it cites an astronomy paper on gender precarity and a pop-science article about drones.\n\nThe stress-test note is correct: the evaluation protocol is the load-bearing flaw. The method itself is coherent and the pipeline is plausible. If the authors define the metric, release the data, add error bars, and test against an existing TCM benchmark, this could become a useful resource. In its current state, it is a work-in-progress report.\n\nWho gets value? Someone working on TCM NLP might scan it for architecture ideas, but not for trustworthy results. It deserves a serious referee only if the evaluation is fixed; as submitted, it would need major revision. I would accept it for peer review with the expectation of heavy revision, not desk-reject it, because the underlying system and data collection are real and a careful reviewer could help the authors turn it into something usable.","headline":"A workmanlike TCM retrieval system whose only evidence for its headline claim is an undefined score with no error bars; the idea is plausible but the evaluation is not.","tokens_in":9339,"tokens_out":2617,"would_cite":false,"duration_ms":22850,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining two-stage RAG retrieval, reranking, and Jieba keyword matching raises TCM case-analysis scores from 36.15 to 37.05, and that this hybrid system produces more accurate LLM-based syndrome differentiation.","keywords":["Traditional Chinese Medicine","syndrome differentiation","retrieval-augmented generation","Jieba segmentation","hybrid retrieval","reranking","large language models","prompt engineering"],"falsifier":"Re-run the same hybrid system (gte-Qwen2-1.5B-instruct embeddings, Jieba matching, gte-passage-ranking-multilingual-base reranking, DeepSeekV2 generation) on a fixed, public set of TCM syndrome-differentiation questions with a defined accuracy metric such as exact syndrome-label agreement; if Jieba's gain over naive RAG disappears or the absolute scores differ wildly, the central claim would be overturned.","tokens_in":8398,"feed_emoji":"🩺","tokens_out":5869,"duration_ms":46887,"temperature":0.7,"pith_summary":"The paper argues that a hybrid retrieval pipeline—dense vector search plus Jieba keyword matching, followed by reranking—measurably improves large language model outputs for Traditional Chinese Medicine syndrome differentiation. It constructs a structured database of over 5,000 scraped TCM clinical cases, then uses retrieved examples inside chain-of-thought prompts to generate JSON-formatted diagnoses with DeepSeekV2. The reported result is a score increase from 36.15 (naive RAG) to 37.05 (RAG plus Jieba), and a model comparison in which DeepSeekV2 scores highest. If the result holds, it offers a practical recipe for making LLMs more credible in domain-specific clinical reasoning.","feed_headline":"Jieba keyword matching lifts TCM retrieval scores to 37.05","feed_subtitle":"The paper reports that adding Jieba to two-stage RAG beats naive RAG on TCM case analysis by 0.90 points.","key_machinery":"The mechanism is a hybrid matching-and-reranking pipeline. A two-stage retriever first uses dense vector similarity (gte-Qwen2-1.5B-instruct) to pull candidate case chunks, then a multilingual reranker (gte-passage-ranking-multilingual-base) reorders them; in parallel, Jieba segmentation converts both query and indexed cases into keyword tokens, and IoU-style lexical matching captures terms that dense vectors miss. The top-ranked case after reranking is inserted as a demonstration into a chain-of-thought prompt, which DeepSeekV2 follows to output a JSON diagnosis. The paper credits the combination—not any single component—for the accuracy gain.","core_discovery":"The central claim is that adding keyword-level matching to semantic retrieval gives LLM-based TCM diagnosis a measurable accuracy boost. Specifically, the system embeds queries and cases with gte-Qwen2-1.5B-instruct, retrieves an initial pool, reranks with gte-passage-ranking-multilingual-base, and supplements the vector results with Jieba segmentation for lexical matches; the best retrieved case is placed into a chain-of-thought prompt, and DeepSeekV2 produces structured JSON answers. The paper reports that this hybrid method scores 37.05 versus 36.15 for naive RAG alone, and that DeepSeekV2 outperforms GPT-4o, Qwen2-72B, GLM-4, and Baidu ERNIE on its evaluation.","pith_inferences":["If the score gap is real, a practical lesson is that lexical matching should not be discarded when adopting RAG; the two cover different failure modes, but this paper does not test whether Jieba alone or reranking alone drives the gain.","A natural next experiment is an ablation that isolates Jieba's contribution from reranking's, together with a defined evaluation metric; without that, the 0.90-point difference is not comparable to any outside benchmark.","The approach could be stress-tested by applying the same hybrid pipeline to an external, scored TCM syndrome-differentiation benchmark where ground-truth labels are public; the paper does not do so."],"forward_implications":["Adding lexically grounded keyword matches to dense retrieval is directly transferable to other Chinese-language medical or domain-specific RAG systems where terminology is highly standardized.","The structured 5,000-case TCM database, with fields for patient details, pathogenesis, syndromes, and notes, becomes a reusable resource for benchmarking future TCM LLMs.","The reported model ranking (DeepSeekV2 above GPT-4o and Qwen2-72B) suggests that smaller, domain-tuned API models can compete with large general models when retrieval and prompting are handled explicitly.","Combining CoT prompt structure with a single high-relevance retrieved example offers a template for explainable, JSON-formatted outputs in clinical reasoning tasks."],"supporting_citations":[{"why":"Supplies the DeepSeekV2 API used as the answer-generation LLM in the system.","marker":"[4]"},{"why":"Cited as the source of the gte-Qwen2-1.5B-instruct embedding model used for semantic vector generation.","marker":"[6]"},{"why":"Provides the retrieval-augmented generation framework that the hybrid pipeline builds on.","marker":"[7]"},{"why":"Supplies Jieba segmentation for keyword extraction and lexical matching.","marker":"[21]"},{"why":"Supplies the gte-passage-ranking-multilingual-base reranker used to reorder initial retrieval results.","marker":"[22]"},{"why":"Supports the reranking strategy as a means of improving retrieval ranking quality.","marker":"[5]"},{"why":"Provides the chain-of-thought prompting method that structures the diagnostic reasoning steps.","marker":"[14]"}],"fun_headline_variants":["Hybrid RAG with Jieba boosts TCM retrieval by 0.90 points","Two-stage RAG plus Jieba beats naive RAG on TCM cases","Keyword-augmented RAG reaches 37.05 in TCM case analysis","Adding lexical match to semantic RAG lifts TCM scores","Jieba-enhanced retrieval outperforms standard RAG in TCM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire reported accuracy improvement rests on an undefined 'Score' metric with no stated test questions, gold-standard answers, or evaluation procedure, so if that score does not measure TCM diagnostic quality, the claimed gain cannot be interpreted.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid RAG with Jieba boosts TCM retrieval by 0.90 points","Two-stage RAG plus Jieba beats naive RAG on TCM cases","Keyword-augmented RAG reaches 37.05 in TCM case analysis","Adding lexical match to semantic RAG lifts TCM scores","Jieba-enhanced retrieval outperforms standard RAG in TCM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000469,"raw_usage":{"total_tokens":2286,"prompt_tokens":849,"completion_tokens":1437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":1337}},"tokens_in":465,"tokens_out":1437,"duration_ms":11087,"temperature":1.0,"reasoning_tokens":1337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:13:38.316556+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same hybrid system (gte-Qwen2-1.5B-instruct embeddings, Jieba matching, gte-passage-ranking-multilingual-base reranking, DeepSeekV2 generation) on a fixed, public set of TCM syndrome-differentiation questions with a defined accuracy metric such as exact syndrome-label agreement; if Jieba's gain over naive RAG disappears or the absolute scores differ wildly, the central claim would be overturned.","supporting_citations":[{"cited_title":"Journal of Physics: Conference Series 1302, 022010 (08 2019).https://doi.org/10.1088/1742-6596/1302/2/022010","cited_arxiv_id":null,"evidence_quote":"Supplies Jieba segmentation for keyword extraction and lexical matching."}],"review_version":1}