Pith. sign in

REVIEW 4 major objections 5 minor 22 references

Traditional Chinese Medicine Case Analysis System for High-Level Semantic Abstraction: Optimized with Prompt and RAG

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that combining two-stage RAG retrieval, reranking, and Jieba keyword matching raises TCM case-analysis scores from 36.15 to 37.05, and that this hybrid system produces more accurate LLM-based syndrome differentiation.

desk verdict A workmanlike TCM retrieval system whose only evidence for its headline claim is an undefined score with no error bars; the idea is plausible but the evaluation is not. read the letter →

arxiv 2411.15491 v1 pith:X4SDQRT4 submitted 2024-11-23 cs.CL

Peng Xu , Hongjin Wu , Jinle Wang , Rongjia Lin , Liwei Tan This is my paper · ORCID
classification cs.CL
keywords TraditionalChineseMedicinesyndromedifferentiationretrieval-augmentedgenerationJiebasegmentationhybridretrievalrerankinglargelanguagemodelspromptengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a hybrid retrieval pipeline—dense vector search plus Jieba keyword matching, followed by reranking—measurably improves large language model outputs for Traditional Chinese Medicine syndrome differentiation. It constructs a structured database of over 5,000 scraped TCM clinical cases, then uses retrieved examples inside chain-of-thought prompts to generate JSON-formatted diagnoses with DeepSeekV2. The reported result is a score increase from 36.15 (naive RAG) to 37.05 (RAG plus Jieba), and a model comparison in which DeepSeekV2 scores highest. If the result holds, it offers a practical recipe for making LLMs more credible in domain-specific clinical reasoning.

What carries the argument

The mechanism is a hybrid matching-and-reranking pipeline. A two-stage retriever first uses dense vector similarity (gte-Qwen2-1.5B-instruct) to pull candidate case chunks, then a multilingual reranker (gte-passage-ranking-multilingual-base) reorders them; in parallel, Jieba segmentation converts both query and indexed cases into keyword tokens, and IoU-style lexical matching captures terms that dense vectors miss. The top-ranked case after reranking is inserted as a demonstration into a chain-of-thought prompt, which DeepSeekV2 follows to output a JSON diagnosis. The paper credits the combination—not any single component—for the accuracy gain.

What would settle it

Re-run the same hybrid system (gte-Qwen2-1.5B-instruct embeddings, Jieba matching, gte-passage-ranking-multilingual-base reranking, DeepSeekV2 generation) on a fixed, public set of TCM syndrome-differentiation questions with a defined accuracy metric such as exact syndrome-label agreement; if Jieba's gain over naive RAG disappears or the absolute scores differ wildly, the central claim would be overturned.

Watch

Extended reading notes

Core claim

The central claim is that adding keyword-level matching to semantic retrieval gives LLM-based TCM diagnosis a measurable accuracy boost. Specifically, the system embeds queries and cases with gte-Qwen2-1.5B-instruct, retrieves an initial pool, reranks with gte-passage-ranking-multilingual-base, and supplements the vector results with Jieba segmentation for lexical matches; the best retrieved case is placed into a chain-of-thought prompt, and DeepSeekV2 produces structured JSON answers. The paper reports that this hybrid method scores 37.05 versus 36.15 for naive RAG alone, and that DeepSeekV2 outperforms GPT-4o, Qwen2-72B, GLM-4, and Baidu ERNIE on its evaluation.

Load-bearing premise

The entire reported accuracy improvement rests on an undefined 'Score' metric with no stated test questions, gold-standard answers, or evaluation procedure, so if that score does not measure TCM diagnostic quality, the claimed gain cannot be interpreted.

Editorial extensions

If this is right

  • Adding lexically grounded keyword matches to dense retrieval is directly transferable to other Chinese-language medical or domain-specific RAG systems where terminology is highly standardized.
  • The structured 5,000-case TCM database, with fields for patient details, pathogenesis, syndromes, and notes, becomes a reusable resource for benchmarking future TCM LLMs.
  • The reported model ranking (DeepSeekV2 above GPT-4o and Qwen2-72B) suggests that smaller, domain-tuned API models can compete with large general models when retrieval and prompting are handled explicitly.
  • Combining CoT prompt structure with a single high-relevance retrieved example offers a template for explainable, JSON-formatted outputs in clinical reasoning tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the score gap is real, a practical lesson is that lexical matching should not be discarded when adopting RAG; the two cover different failure modes, but this paper does not test whether Jieba alone or reranking alone drives the gain.
  • A natural next experiment is an ablation that isolates Jieba's contribution from reranking's, together with a defined evaluation metric; without that, the 0.90-point difference is not comparable to any outside benchmark.
  • The approach could be stress-tested by applying the same hybrid pipeline to an external, scored TCM syndrome-differentiation benchmark where ground-truth labels are public; the paper does not do so.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes the construction of a Traditional Chinese Medicine (TCM) clinical case database (over 5,000 cases scraped from public web sources) and a retrieval-augmented generation (RAG) pipeline for syndrome-differentiation tasks. The pipeline uses semantic vector retrieval (gte-Qwen2-1.5B-instruct), Jieba keyword matching, a reranker (gte-passage-ranking-multilingual-base), and DeepSeekV2 as the generator, with chain-of-thought style prompts producing structured JSON outputs. The central claim, stated in the abstract and Section 4.2, is that combining two-stage retrieval with Jieba keyword matching 'significantly enhanced the accuracy of model outputs.' The empirical support consists of two tables of single 'Score' values, but the metric, test set, scoring rubric, and experimental procedure are never defined.

Significance. If the claims were substantiated, the dataset and hybrid retrieval pipeline could be a useful engineering contribution for TCM natural language processing, an area with few openly available structured clinical corpora. The system sensibly combines established components (RAG, reranking, keyword matching), and the dataset-construction effort addresses a real need. However, the paper provides no valid quantitative evidence for its central claim: the reported scores are undefined, the comparison in Section 4.2 is mislabeled, and no error bars or significance tests are given. As submitted, the contribution is unverified; a proper evaluation on a defined benchmark would be needed before the results could be considered reliable.

major comments (4)
  1. [Section 4.2, Tables 1 and 2] The 'Score' metric in Tables 1 and 2 is never defined, and the test questions, dataset split, and scoring procedure are not described. Without these, the 0.90-point difference between 36.15 and 37.05 in Table 1 cannot be interpreted, and the abstract's claim of 'significantly enhanced accuracy' is unsupported. This is a load-bearing issue because the entire contribution rests on this comparison.
  2. [Section 4.2, text vs. Table 1] The prose in Section 4.2 compares 'Jieba' with 'overlap window,' but Table 1 labels the rows 'Naive RAG' and 'Naive RAG + Jieba.' This mismatch makes it impossible to know which system configurations are being compared and whether the reported gain is due to Jieba, RAG, or their combination. The text also treats 'RAG' and 'Jieba' as competing methods, while the table suggests the comparison is between RAG with and without Jieba.
  3. [Section 4 (general)] No error bars, confidence intervals, or significance tests are reported anywhere in Section 4. Given the unknown number of test cases and the small reported difference (0.90 points), the observed improvement could be run-to-run noise or the effect of a single test question. Additionally, Table 2 reports base-model scores in the 24–27 range while Table 1 reports 36–37, implying different tasks or different score scales; this prevents any cross-table interpretation of absolute values.
  4. [Sections 3.2 and 4.1] The qualitative examples in Figure 4 and Section 4.1 purport to show that RAG improves diagnostic reasoning, but no quantitative measure supports this. The chain-of-thought discussion in Section 3.2 similarly presents only a single illustrative example. These claims are not backed by experimental evidence and cannot substitute for the missing evaluation.
minor comments (5)
  1. [Throughout] There are numerous typos and formatting errors, such as 'T able 1,' missing spaces between words (e.g., 'theBaidu_ERNIE_Speed_128KAPI'), and inconsistent spellings of model names. The paper needs thorough proofreading.
  2. [References] Reference [13] is 'Gender and Precarity in Astronomy' and reference [1] is about line spacing in LaTeX; neither appears relevant to the claims they support. The authors should verify that all references are appropriate and correctly cited.
  3. [Section 3.1] Section 3.1 gives a URL for the dataset (https://cccl-tcm.cacm.org.cn/article?lang=zh) but does not specify licensing, access conditions, or whether the processed dataset is actually downloadable. This should be clarified.
  4. [Section 3.1] The paper claims 'over 5,000 TCM clinical cases' but provides no statistics about the corpus (e.g., number of syndromes, distribution of case lengths, or sources), so the reader cannot assess the dataset's coverage or representativeness.
  5. [Figures] The figure captions are minimal and sometimes do not explain the claimed workflow; for example, Figure 3's caption does not describe the OCR and text-cleaning steps mentioned in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical engineering results, not derivations that reduce to their own inputs.

full rationale

The paper makes no formal derivation whose conclusion is identical to its premises. The central quantitative claim—that adding Jieba keyword matching to a two-stage RAG pipeline improves scores (36.15 vs 37.05 in Table 1)—is an empirical comparison, not a set of equations that reduce to themselves. No fitted parameter is renamed as a prediction; no uniqueness theorem is imported from the authors' prior work; no ansatz is smuggled in via self-citation. The references point to third-party tools and models (FAISS, Jieba, gte-Qwen2, DeepSeek, rerankers) and not to the authors' own prior results. The main weakness is experimental transparency: 'Score' in Tables 1 and 2 is never defined, the test set and rubric are undisclosed, and the 0.90-point gap has no error bars or significance test, so 'significantly enhanced' is not statistically supported. However, that is a validity and reproducibility concern, not circularity: an underspecified evaluation cannot be shown to be equivalent to its inputs by construction. The two tables also appear to use different tasks or scales, further undermining interpretation, but again this does not constitute a definitional circularity. Accordingly the circularity score is 0; the correctness risk should be addressed through a defined metric and external validation, not through a circularity finding.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted numeric parameters and no invented entities. The load-bearing non-mathematical assumptions are that the LLM-based cleaning preserves clinical meaning, that the web-scraped cases are reliable, and that the undisclosed scoring scheme used in Tables 1 and 2 measures diagnostic accuracy. These assumptions are stated nowhere and are not validated.

assumptions (3)
  • ad hoc to paper The 'score' reported in Tables 1 and 2 is a valid, consistent measure of TCM diagnostic reasoning quality.
    The metric is never defined; the entire performance comparison rests on this unstated assumption.
  • domain assumption Web-scraped cases from 360doc and other sites provide reliable ground-truth TCM clinical content after LLM-based cleaning.
    Data quality underpins the database and all retrieval and answer evaluations, as described in Section 3.1.
  • domain assumption LLM cleaning and structuring preserves the clinical meaning of the original cases.
    The pipeline uses Baidu ERNIE to filter and structure cases, but no manual validation of the structured output is reported in Section 3.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Traditional Chinese Medicine Case Analysis System for High-Level Semantic Abstraction: Optimized with Prompt and RAG." pith.science (2026). https://pith.science/paper/X4SDQRT4

@misc{pith2026241115491,
  author       = {Pith},
  title        = {Pith review of: Traditional Chinese Medicine Case Analysis System for High-Level Semantic Abstraction: Optimized with Prompt and RAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X4SDQRT4}},
  note         = {Machine review of arXiv:2411.15491}
}
abstract

This paper details a technical plan for building a clinical case database for Traditional Chinese Medicine (TCM) using web scraping. Leveraging multiple platforms, including 360doc, we gathered over 5,000 TCM clinical cases, performed data cleaning, and structured the dataset with crucial fields such as patient details, pathogenesis, syndromes, and annotations. Using the $Baidu\_ERNIE\_Speed\_128K$ API, we removed redundant information and generated the final answers through the $DeepSeekv2$ API, outputting results in standard JSON format. We optimized data recall with RAG and rerank techniques during retrieval and developed a hybrid matching scheme. By combining two-stage retrieval method with keyword matching via Jieba, we significantly enhanced the accuracy of model outputs.

Figures

Figures reproduced from arXiv: 2411.15491 by the authors.

Figure 1
Figure 1. illustrates the difference between a prompt without Chain-of-Thought [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 1
Figure 1. An example of constructing chain-of-thought prompts in the TCM syndrome differentiation task involves integrating reasoning pathways into the diagnostic process. Compared with the original prompt, the chain-of-thought prompt adds the correspond￾ing reasoning path for extracting clinical information, inferring pathomechanisms, and determining syndromes before outputting the final diagnosis, ensuring a systematic and … view at source ↗
Figure 2
Figure 2. – Two-Stage Retrieval Framework. The RAG framework employs a two￾stage retrieval process to extract and rank relevant information from a pre￾constructed TCM knowledge base. First, both the user query and the in￾dexed knowledge are vectorized, with similarity-based ranking performed using tools such as FAISS (Facebook AI Similarity Search). The system retrieves a pool of knowledge chunks in the initial stage based on… view at source ↗
Figures from the paper (3 more)
Figure 2
Figure 2. Figure 2: Illustration of a two-stage retrieval process in a Retrieval-Augmented Gen￾eration (RAG) framework for TCM diagnosis, integrating user queries with external knowledge to enhance reasoning and output accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_2.png]
Figure 3
Figure 3. Figure 3: Illustration of a preprocessing pipeline for structured and unstructured data, integrating OCR, text cleaning, jieba chunking, text segmentation, and vectorization for creating a searchable hybrid database [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]
Figure 4
Figure 4. Figure 4: Comparison of Diagnostic Reasoning with and without Retrieval-Augmented Generation (RAG) in Traditional Chinese Medicine Syndrome Differentiation [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 8 canonical work pages

  1. [1]

    [Online], https://www.popsci.com/article/ technology/how-will-drones-change-sports/

    Line spacing in latex documents. [Online], https://www.popsci.com/article/ technology/how-will-drones-change-sports/

    work page

  2. [2]

    [Online], https:// zilliz.com/blog/hybrid-search-with-milvus

    Batifol, S.: Getting started with hybrid search with milvus. [Online], https:// zilliz.com/blog/hybrid-search-with-milvus

    work page

  3. [3]

    [Online], https://medium.com/@csakash03/ hybrid-search-is-a-method-to-optimize-rag-implementation-98d9d0911341

    Csakash: Hybrid search a method to optimize rag im- plementation. [Online], https://medium.com/@csakash03/ hybrid-search-is-a-method-to-optimize-rag-implementation-98d9d0911341

    work page

  4. [4]

    arXiv e-prints arXiv:2405.04434 (May 2024)

    DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Yang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J.L., Liang, J., Guo, J., Ni, J., Li, J., Chen, J., Yuan, J., Qiu, J., Song, ...

    arXiv doi:10.48550/arxiv.2405.04434 2024

  5. [5]

    arXiv e-prints arXiv:2406.12433 (Jun 2024).https://doi.org/10.48550/arXiv

    Gao,J.,Chen,B.,Zhao,X.,Liu,W.,Li,X.,Wang,Y.,Zhang,Z.,Wang,W.,Ye,Y., Lin, S., Guo, H., Tang, R.: LLM-enhanced Reranking in Recommender Systems. arXiv e-prints arXiv:2406.12433 (Jun 2024).https://doi.org/10.48550/arXiv. 2406.12433

    arXiv doi:10.48550/arxiv.2406.12433 2024

  6. [6]

    arXiv e-prints arXiv:2409.12186 (Sep 2024)

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., Yang, A., Men, R., Huang, F., Ren, X., Ren, X., Zhou, J., Lin, J.: Qwen2.5-Coder Technical Report. arXiv e-prints arXiv:2409.12186 (Sep 2024). https://doi.org/10.48550/arXiv.2409.12186

    arXiv doi:10.48550/arxiv.2409.12186 2024

  7. [7]

    arXiv e-prints arXiv:2005.11401 (May 2020)

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv e-prints arXiv:2005.11401 (May 2020). https://doi.org/10.48550/arXiv.2005.11401

    arXiv doi:10.48550/arxiv.2005.11401 2005

  8. [8]

    Pharmacological Research 178, 106185 (2022)

    Li, X., Ren, J., Zhang, W., Zhang, Z., Yu, J., Wu, J., Sun, H., Zhou, S., Yan, K., Yan, X., Wang, W.: Ltm-tcm: A comprehensive database for the linking of tradi- tional chinese medicine with modern medicine at molecular and phenotypic levels. Pharmacological Research 178, 106185 (2022). https://doi.org/https://doi. 14 P. Xu et al. org/10.1016/j.phrs.2022....

    arXiv 2022

Show all 22 references
  1. [9]

    Bulletin of the Medical Library Association 78, 271–82 (Jul 1990)

    Perry, C.A.: Knowledge bases in medicine: a review. Bulletin of the Medical Library Association 78, 271–82 (Jul 1990)

    1990

  2. [10]

    arXiv e-prints arXiv:2203.10839 (Mar 2022)

    Ren, M., Huang, H., Zhou, Y., Cao, Q., Bu, Y., Gao, Y.: TCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Processing. arXiv e-prints arXiv:2203.10839 (Mar 2022). https://doi.org/10.48550/arXiv.2203. 10839

    2022 doi arXiv

  3. [11]

    arXiv e-prints arXiv:2408.04948 (Aug 2024)

    Sarmah, B., Hall, B., Rao, R., Patel, S., Pasquali, S., Mehta, D.: HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction. arXiv e-prints arXiv:2408.04948 (Aug 2024). https://doi.org/10.48550/arXiv.2408.04948

    2024 doi arXiv

  4. [12]

    arXiv e-prints arXiv:1904.09223 (Apr 2019).https://doi.org/10.48550/arXiv

    Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., Wu, H.: ERNIE: Enhanced Representation through Knowledge Integration. arXiv e-prints arXiv:1904.09223 (Apr 2019).https://doi.org/10.48550/arXiv. 1904.09223

    1904 doi arXiv

  5. [13]

    In: Journées 2022 de la Société Française d’Astronomie & d’Astrophysique (3 2023)

    Webb, N.A., Bot, C., Charpinet, S., Contini, T., Jouve, L., Meheut, H., Mei, S., Mosser, B., Soucail, G.: Gender and Precarity in Astronomy. In: Journées 2022 de la Société Française d’Astronomie & d’Astrophysique (3 2023)

    2022

  6. [14]

    arXiv e-prints arXiv:2201.11903 (Jan 2022).https://doi.org/10.48550/ arXiv.2201.11903

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv e-prints arXiv:2201.11903 (Jan 2022).https://doi.org/10.48550/ arXiv.2201.11903

    2022 doi arXiv

  7. [15]

    arXiv e-prints arXiv:2301.05578 (Jan 2023).https: //doi.org/10.48550/arXiv.2301.05578

    Weisz, J.D., Muller, M., He, J., Houde, S.: Toward General Design Principles for Generative AI Applications. arXiv e-prints arXiv:2301.05578 (Jan 2023).https: //doi.org/10.48550/arXiv.2301.05578

    2023 doi arXiv

  8. [16]

    arXiv e-prints arXiv:2308.03549 (Aug 2023)

    Yang, S., Zhao, H., Zhu, S., Zhou, G., Xu, H., Jia, Y., Zan, H.: Zhongjing: Enhanc- ing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue. arXiv e-prints arXiv:2308.03549 (Aug 2023). https://doi.org/10.48550/arX...

    2023 doi arXiv

  9. [17]

    arXiv e-prints arXiv:2406.01126 (Jun 2024)

    Yue, W., Wang, X., Zhu, W., Guan, M., Zheng, H., Wang, P., Sun, C., Ma, X.: TCMBench: A Comprehensive Benchmark for Evaluating Large Language Mod- els in Traditional Chinese Medicine. arXiv e-prints arXiv:2406.01126 (Jun 2024). https://doi.org/10.48550/arXiv.2406.01126

    2024 doi arXiv

  10. [18]

    arXiv e-prints arXiv:2403.16056 (Mar 2024).https://doi.org/10.48550/arXiv.2403.16056

    Zhang, H., Wang, X., Meng, Z., Chen, Z., Zhuang, P., Jia, Y., Xu, D., Guo, W.: Qibo: A Large Language Model for Traditional Chinese Medicine. arXiv e-prints arXiv:2403.16056 (Mar 2024).https://doi.org/10.48550/arXiv.2403.16056

    2024 doi arXiv

  11. [19]

    BioMed Research International 2022, 1–7 (01 2022)

    Zhang, P., Shen, S., Deng, W., Mao, S., Wang, Y.: The construction model of the tcm clinical knowledge coding database based on knowledge organization. BioMed Research International 2022, 1–7 (01 2022). https://doi.org/10.1155/2022/ 2503779

    2022 doi

  12. [20]

    Evidence-based complementary and alternative medicine : eCAM2022, 1679589 (2022)

    Zhang, T., Huang, Z., Wang, Y., Wen, C., Peng, Y., Ye, Y.: Information extraction from the text data on traditional chinese medicine: A review on tasks, challenges, and methods from 2010 to 2021. Evidence-based complementary and alternative medicine : eCAM2022, 1679589 (2022)

    2022

  13. [21]

    Journal of Physics: Conference Series 1302, 022010 (08 2019).https://doi.org/10.1088/1742-6596/1302/2/022010

    Zhang, X., Wu, P., Cai, J., Wang, K.: A contrastive study of chinese text segmen- tation tools in marketing notification texts. Journal of Physics: Conference Series 1302, 022010 (08 2019).https://doi.org/10.1088/1742-6596/1302/2/022010

    2019 doi

  14. [22]

    arXiv preprint arXiv:2407.19669 (2024)

    Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., et al.: mgte: Generalized long-context text representation and Optimizing TCM Case Analysis with Prompt Engineering and RAG 15 reranking models for multilingual text retrieval. a...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.