{"id":"25170b87-2502-4a7a-9365-522d72a8e719","arxiv_id":"2501.04635","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A retrieval-augmented generation system using BGE-M3 and a reranker improves LLM accuracy on Chinese QA benchmarks, especially for finance and law questions.","lead":"This paper builds a question-answering system that searches Chinese Wikipedia and a legal database, then feeds the top results to a large language model. The system improves the model's accuracy on Chinese multiple-choice benchmarks and helps people without finance backgrounds answer finance questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unvalidated BGE-M3 cosine-similarity answer-label mapping is the load-bearing step: all post-refinement accuracy numbers, and possibly the TTQA headline, depend on it, and no human agreement check is reported.","rationale":"The central claim—RAG improves LLM accuracy—is supported in broad strokes by a consistent across-the-board improvement on TTQA for 14 models, and by the internal consistency of the pre-/post-refinement gain arithmetic on TMMLU+ finance/insurance. That reduces the risk that the paper is reporting a null effect as positive. The single most load-bearing weakness is therefore not the direction of the effect but the measurement instrument: the cosine-similarity label mapping is used to convert free-form generations into accuracies, and it is unvalidated. This is exactly the reader's weakest_assumption, so I agree. The reader's other concerns (selective TMMLU+ reporting, confounded human study) are secondary: they affect the generality and framing of the claims, not the core TTQA result. Because the qualitative finding likely survives even a modest mislabeling rate, and the flaws are fixable with additional evaluation, the reader's CONDITIONAL verdict remains appropriate. I would condition acceptance on the annotation check and on the release of evaluation scripts and prompts.","tokens_in":8345,"tokens_out":6786,"duration_ms":57926,"concrete_test":"Select the 103 TTQA responses from Taiwan-LLM-8x7B-DPO w/ RAG and, if available, the post-refinement finance/insurance responses. Have two independent annotators who are native Traditional Chinese speakers read each raw generated response and select the intended option label. Measure inter-annotator agreement (Cohen's kappa) and disagreement between annotator labels and the BGE-M3 cosine-similarity labels. If the mapping error rate exceeds ~5%, or if errors are systematically biased toward one option (e.g., always picking the longest option), recompute the reported accuracies using the human labels. If the recomputed TTQA accuracy gap (57.28 vs. 88.35) or the finance gain (e.g., 21.24 for Breeze) moves by more than a few percentage points, the paper's quantitative claims need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the automatic answer-label mapping described in Section IV's 'Refine user prompt' paragraph. After stripping categorical labels, the LLM's free-form response is embedded with BGE-M3, cosine similarity to each stripped option is computed, and the argmax is taken as the predicted label. This mapping produces the post-refinement accuracy numbers on the TMMLU+ finance/insurance subsets—the largest reported RAG gains—and the paper offers no validation against human annotation, no error rate, no tie-breaking rule, and no threshold for confident assignment. The single illustrative example (0.973 vs. 0.771, 0.663, 0.775) shows a clear margin, but it does not establish reliability across hundreds of responses. A generated response can mention several options or hedge; cosine similarity over the whole text can then select the wrong option, or be sensitive to wording length. Because every accuracy percentage in the strongest claim is derived from these labels, an unmeasured mislabeling rate directly biases the headline improvements (e.g., 57.28→88.35 on TTQA, and 16–21 point gains on finance). If the TTQA numbers in Table I were produced by the pre-refinement constrained protocol, the mapping concern applies directly only to the finance/insurance numbers; but the paper never documents the TTQA label extraction, so the evaluation protocol for the headline claim is underspecified either way. The paper also provides no confidence intervals or significance tests; with n=103 (TTQA) and n=160/855 (finance/insurance), sampling noise alone could shift some reported gaps. The human study, while confounded, is not the main pillar; the automatic evaluation is.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper builds a Retrieval-Augmented Generation (RAG) question-answering system that combines BGE-M3 dense retrieval, BGE-reranker re-ranking, and source corpora from Chinese Wikipedia and Lawbank legal texts. It evaluates 14 large language models on the TTQA benchmark and on finance/insurance subsets of TMMLU+, reporting large accuracy gains when RAG is used, e.g., Taiwan-LLM-8x7B-DPO rising from 57.28% to 88.35% on TTQA, and further gains on finance/insurance after a prompt-refinement step. A human study with 20 non-expert participants reports an increase in mean score from 28.75 to 85.25 when participants answer the same 20 finance questions after receiving RAG-generated reference information. The main contributions claimed are enhanced LLM accuracy on knowledge-intensive tasks and a locally deployable RAG architecture that improves data privacy.","tokens_in":8614,"tokens_out":4083,"duration_ms":37187,"significance":"If the reported effects are real, the paper provides a useful empirical demonstration that off-the-shelf dense retrieval and re-ranking components can improve LLM performance on Traditional Chinese knowledge tasks, and that domain-specific retrieval sources can matter for specialized domains. The paper has concrete strengths: it uses standard public benchmarks (TTQA, TMMLU+), evaluates multiple LLMs, and reports specific accuracy numbers. However, the central evaluation pipeline rests on an unvalidated automatic answer-label mapping, the TMMLU+ domain experiments are selected post hoc, and the human study lacks controls for learning and order effects. These issues mean the quantitative claims are not yet established at the level needed for publication; the paper is a promising empirical study that requires substantial additional validation.","major_comments":[{"comment":"The automatic answer-label mapping introduced in the 'Refine user prompt' paragraph is load-bearing: every post-refinement accuracy number on the TMMLU+ finance/insurance subsets (e.g., gains of 16.24%, 21.24%, 8.75%, and 4.37% for financial banking) is derived by taking the argmax of BGE-M3 cosine similarity between the LLM-generated text and the stripped option strings. The paper reports no validation of this mapping against human annotation, no error rate, no tie-breaking rule, and no confidence threshold. Because a free-form generated response can mention several options or hedge, the cosine-similarity argmax can mislabel answers and directly inflate or deflate the reported accuracy gains. Please validate the mapping on a random sample (e.g., 100-200 items) with human labels, report agreement, and clarify how ties and responses that do not correspond to any option are handled.","section":"IV (Experiments), 'Refine user prompt' paragraph"},{"comment":"The TMMLU+ finance/insurance experiments are introduced only after the authors observe that general Wikipedia RAG yields below-50% accuracy across all 66 TMMLU+ topics. This post-hoc subset selection means the reported Lawbank gains (13.74-16.25% for banking; 7.25-15.09% for insurance) are selected from the exact topics where the contrast is expected to be largest, which biases the domain-specificity claim. The paper should either pre-register the domain hypothesis, report results for all 66 topics, or otherwise correct for multiple comparisons; without this, the domain-gain result is not a confirmatory finding.","section":"IV (Experiments), paragraph beginning 'Observations from previous experiments'"},{"comment":"The two-stage human study has no control for learning or answer-copying: the same 20 participants answer the same 20 finance questions first without and then with RAG-generated reference material, in a fixed order, with no counterbalancing, washout, or control group. The mean increase from 28.75 to 85.25 could reflect memory of the pre-test, practice effects, or demand characteristics rather than the value of the RAG references. The paper also reports no confidence intervals or significance tests, so the claim that RAG-generated reference information 'significantly enhanced' participant performance is unsupported. At minimum, report paired pre/post statistics (e.g., a paired t-test or Wilcoxon signed-rank test with an effect size) and either redesign the experiment or explicitly discuss and mitigate the order confound.","section":"IV (Experiments), human study (Figures 11-12)"},{"comment":"The protocol used to obtain the TTQA accuracy numbers is not documented. The text states that accuracy is computed by comparing 'auto-generated labels' with ground truth, but it does not specify whether TTQA answers were constrained multiple-choice outputs or mapped through the same BGE-M3 cosine-similarity procedure described later in Section IV. Since the headline result (Taiwan-LLM-8x7B-DPO rising from 57.28% to 88.35% with RAG) depends on this label extraction, the paper must specify the exact extraction method for TTQA and, if it is the cosine-similarity mapping, subject it to the same human validation requested above.","section":"IV (Experiments), Table I and surrounding text"}],"minor_comments":[{"comment":"Table I lists 13 models while the text states that 14 LLMs were used; please reconcile this discrepancy and indicate which model is omitted (e.g., TAIDE-LX-7B-Chat appears in the TMMLU+ experiments but not in Table I).","section":"IV, Table I"},{"comment":"The paper states that 1,377,100 Wikipedia articles had fewer than 8,192 tokens, but it does not describe how the remaining articles that exceed the token limit were chunked or handled; please provide this detail for reproducibility.","section":"III (Approach), Section B"},{"comment":"The figures (Figures 3-12) contain no error bars or confidence intervals; adding them, particularly for the accuracy comparisons across models, would improve interpretability.","section":"Throughout"},{"comment":"The phrase 'as Figure 3 showed' and similar tense constructions appear inconsistently; use 'Figure 3 shows' for consistency, and correct the caption 'Refine user prompt for multi-choices question' to 'multiple-choice'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the unvalidated BGE-M3 cosine-similarity answer-label mapping is valid and is the primary reason for major revision: it directly affects the post-refinement TMMLU+ numbers and potentially the TTQA headline. The paper would also benefit from a reproducibility statement, including code, prompts, and seed information, since none are currently provided. I do not see circularity or fabrication; the issues are about validation and experimental design, both of which are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent but unremarkable RAG system evaluation for Traditional Chinese legal/finance QA. The central claim—RAG with dense retrieval and reranking beats no-RAG—is probably true, and the TTQA gains are consistent across all 13 reported models, which is a genuinely strong pattern. But the exact numbers should not be trusted until the answer-label mapping is validated.\n\nWhat's new is narrow: the specific combination of BGE-M3, BGE-reranker, FAISS, Chinese Wikipedia plus Lawbank, evaluated on TTQA and the TMMLU+ finance/insurance subsets. No new method, but new benchmark numbers for a less-covered language/domain pairing, and the local-deployment privacy story is reasonable. I also credit the authors for reporting that RAG hurts or doesn't help on many TMMLU+ topics; that is an honest negative result.\n\nThe main soft spot is the automatic label extraction. The paper strips labels from options, embeds the model's free-form answer and each option with BGE-M3, and takes the argmax cosine similarity as the predicted answer. There is no human agreement check, no tie-breaking rule, no confidence threshold, and no error rate. That is load-bearing because it produces every post-refinement accuracy number, including the headline TTQA gains (if those went through the same pipeline; the paper doesn't document TTQA's label extraction, so it's underspecified either way). The single illustrative example has a clear margin, but a few hundred responses are not the same.\n\nThe second soft spot is the post hoc TMMLU+ subset selection. The authors first ran all 66 topics, saw poor RAG performance, then focused on finance and insurance with Lawbank. That is cherry-picking unless pre-registered or justified with a held-out protocol. The reported 16–21 point gains on those subsets are therefore overstated as evidence of domain-specific RAG value.\n\nMinor items: no confidence intervals or significance tests (n=103 for TTQA, n=160/855 for the subsets), no BM25 or sparse retrieval baseline, and the human study is confounded by repeated questions and no control group. Those are minor because the automatic evaluation is the main pillar.\n\nWho is this for? Practitioners building private Chinese QA systems will get the most value, and the paper deserves a serious referee. I'd send it to review but require the authors to validate the label mapping against human annotation, report variability, and make prompts and code available.\n\nRecommendation: engage with it, but treat the reported gains as provisional until the evaluation protocol is nailed down.","headline":"Straightforward RAG engineering for Chinese legal/finance QA with an unvalidated answer-scoring step; the gains are directionally plausible but the exact numbers should be treated as provisional.","tokens_in":9206,"tokens_out":3055,"would_cite":false,"duration_ms":28478,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding BGE-M3 dense retrieval and BGE-reranker re-ranking to a RAG pipeline raises a Taiwanese LLM's accuracy on a 103-question trivia benchmark from 57.28% to 88.35%, and domain-specific legal retrieval beats general Wikipedia on finance…","keywords":["Retrieval Augmented Generation","Dense Vector Search","Re-ranking","Large Language Model","Evaluation","TTQA","TMMLU+","Traditional Chinese question answering"],"falsifier":"Take a random sample of, say, 100 items from TTQA and the TMMLU+ finance and insurance subsets, have two human annotators judge the LLM's free-form answers, and compare their labels with the cosine-similarity auto-labels; large disagreement would mean the reported accuracy gains are artifacts of the evaluation pipeline rather than true improvements in question answering.","tokens_in":8163,"feed_emoji":"🔍","tokens_out":16211,"duration_ms":142313,"temperature":0.7,"pith_summary":"This paper attempts to show that a Retrieval-Augmented Generation (RAG) system built from BGE-M3 dense retrieval, BGE-reranker re-ranking, and a vector index can make large language models substantially better at answering Traditional Chinese multiple-choice questions. On the 103-question TTQA (Taiwanese Trivia Question Answering) benchmark, the reported accuracy of Taiwan-LLM-8x7B-DPO rises from 57.28% to 88.35% when retrieved reference passages are added, and ChatGPT 3.5 rises from 74.76% to the same 88.35%. On the finance and insurance subsets of the 66-topic TMMLU+ suite, retrieving from the specialized Lawbank legal corpus beats retrieving from Chinese Wikipedia for every tested model, with gains of up to 16.25 and 15.09 percentage points before prompt refinement. A 20-participant human study on 20 finance questions reports average scores rising from 28.75 to 85.25 when the system supplies reference sentences. The payoff the paper argues for is that a locally hosted RAG pipeline can reduce hallucinations, provide up-to-date or domain-specific knowledge, and avoid sending private data to commercial APIs.","feed_headline":"Retrieval step lifts AI quiz accuracy from 57% to 88%","feed_subtitle":"Dense retrieval plus reranking lifts a local model to ChatGPT-level scores, no cloud needed.","key_machinery":"The load-bearing mechanism is a retrieval pipeline built from two models: BGE-M3, a multilingual embedding model that maps queries and documents to vectors for dense, sparse, and multi-vector retrieval with up to 8,192 tokens of context, and BGE-reranker, a cross-encoder that re-scores retrieved passages by query relevance instead of raw similarity. A vector index stores the embeddings for fast similarity search over 1.38 million Wikipedia articles and 6,193 Lawbank legal clauses. The final answer is produced by feeding the top re-ranked passages to the LLM as reference knowledge; for evaluation, the LLM's free-form answer is mapped to a choice by taking the option with the highest BGE-M3 cosine similarity to the generated text.","core_discovery":"The central claim is that combining a dense retriever with a relevance reranker inside a RAG pipeline improves LLM question answering enough to matter in practice, and that the retrieval source is a decisive factor. The paper reports that every model listed for TTQA scores higher with RAG than without, that domain-specific Lawbank retrieval outperforms general Wikipedia retrieval on finance and insurance questions for all four models tested, and that removing format restrictions plus scoring answers by BGE-M3 cosine similarity adds further gains. It also claims the two-stage human experiment shows RAG-generated reference material is highly effective for people without domain background. The paper's stated contributions are enhanced LLM capability and data privacy through local deployment.","pith_inferences":["Going beyond the paper, the evaluation pipeline itself deserves scrutiny: because BGE-M3 both finds the evidence and scores the answer-option match, the reported gains are not separable from that embedding model's abilities.","A testable extension the paper does not run is a chunking ablation on Lawbank, comparing clause-level chunks against section-level chunks on the same finance questions.","The human experiment generalizes only to lay users; an expert-participant version would test whether RAG assistance still helps when the reader already knows the domain.","If the mechanism is general, the same recipe should transfer to any specialized corpus with clean unit boundaries, such as contracts or medical guidelines, which is a direct way to test the Lawbank result."],"forward_implications":["Reported TTQA results imply that attaching this retrieval pipeline improves every tested LLM, with the largest absolute gain for weaker models and ChatGPT 3.5 reaching the same top score.","Reported TMMLU+ results imply that for specialized finance and insurance questions, the retrieval corpus matters more than the model: Lawbank beats Chinese Wikipedia for all four models tested.","The architecture's local deployment is a direct consequence: all components (index, reranker, and LLM) run on local infrastructure, so private documents never leave the system.","The prompt-refinement step implies that answer-format restrictions suppress accuracy, so the evaluation harness is part of the measured performance."],"supporting_citations":[{"why":"It supplies the retrieval-augmented generation framework that combines retrieval with generation and is the paper's base architecture.","marker":"[4]"},{"why":"It supplies the BGE-M3 embedding model used for dense retrieval and for cosine-similarity matching between generated answers and options, together with the BGE-reranker the paper pairs with it.","marker":"[6]"},{"why":"It provides the TC-Eval suite and the 103-question TTQA benchmark used for the main accuracy comparison.","marker":"[7]"},{"why":"It provides the TMMLU+ benchmark with 66 topics and the finance and insurance subsets used for the domain-specific experiments.","marker":"[8]"},{"why":"It provides the vector index used to search over 1.38 million Wikipedia entries and 6,193 Lawbank clauses.","marker":"[9]"},{"why":"It introduces the Taiwan-LLM model family whose reported TTQA accuracy rises from 57.28% to 88.35% with RAG.","marker":"[10]"},{"why":"It introduces the Breeze-7B model that reports the best TMMLU+ accuracy and serves as a central comparison point.","marker":"[11]"}],"fun_headline_variants":["RAG boosts AI quiz scores from 57% to 88%","Local RAG model matches cloud AI on finance quiz","Dense retrieval plus reranking lifts LLM accuracy","No cloud needed: local RAG beats baseline on quiz"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the automatic step that turns each model's free-form answer into a multiple-choice label—by picking the option most similar to the generated text under the embedding model—produces the same results a human grader would, and the paper does not test that agreement.","fun_headline_variants_meta":{"raw":{"variants":["RAG boosts AI quiz scores from 57% to 88%","Local RAG model matches cloud AI on finance quiz","Dense retrieval plus reranking lifts LLM accuracy","No cloud needed: local RAG beats baseline on quiz"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3273,"prompt_tokens":958,"completion_tokens":2315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2247}},"tokens_in":574,"tokens_out":2315,"duration_ms":16329,"temperature":1.0,"reasoning_tokens":2247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:26:42.213526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 100 items from TTQA and the TMMLU+ finance and insurance subsets, have two human annotators judge the LLM's free-form answers, and compare their labels with the cosine-similarity auto-labels; large disagreement would mean the reported accuracy gains are artifacts of the evaluation pipeline rather than true improvements in question answering.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the vector index used to search over 1.38 million Wikipedia entries and 6,193 Lawbank clauses."}],"review_version":1}