{"id":"dfb8262d-2242-4c97-a398-55a0b16f69ff","arxiv_id":"2502.06572","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A knowledge-guided pipeline uses DeepSeek V3 to generate and self-verify 50K Chinese legal reasoning examples, and fine-tuned Qwen-2.5 models (0.5B-3B) into LawGPT, which outperforms prior legal-specific LLMs on four LawBench-style tasks.","lead":"LawGPT builds a 50K-example Chinese legal reasoning dataset by having DeepSeek V3 generate, correct, and verify problems under guidance from legal documents, then fine-tunes small Qwen-2.5 models on it. The resulting 3B model beats prior legal-specific LLMs and approaches a 671B model on four LawBench-style tasks, at much lower cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No contamination check: the 50K synthetic set is generated from legal corpora that overlap the LawBench test sources, so training/eval leakage could explain LAWGPT's gains; a dedup and retrain test is needed.","rationale":"The reader's weakest assumption focused on self-verification of legal correctness by the generator model. My concern is adjacent but distinct: even if every generated answer is legally correct, the evaluation is invalid if the synthetic training data overlaps the test distribution. The paper's knowledge base consists of public criminal and civil legal documents, and the four evaluation tasks come from the same public Chinese legal sources, so leakage is a plausible failure mode. The reader did mention contamination analysis as a missing piece in the rationale, but not as the primary load-bearing assumption. I keep the CONDITIONAL verdict because the paper is otherwise well-structured with reproducible code and clear experiments; however, the condition should explicitly require an overlap analysis and, if leakage is found, a retrained model on deduplicated data. If the overlap test reveals substantial leakage, the verdict should move to REJECT; if it clears, the central claim is substantially supported.","tokens_in":10472,"tokens_out":3349,"duration_ms":100034,"concrete_test":"Compute near-duplicate overlap between the released 50K synthetic training set and the LawBench test sets for Tasks #1–#4 using 13-gram MinHash/Jaccard similarity, flagging any pair above a threshold (e.g., 0.8). If overlaps exist, retrain LAWGPT on the deduplicated subset and re-evaluate; if average performance drops by more than a few points, the reported gains are at least partly due to leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that fine-tuning on 50K synthetic examples improves general legal reasoning, not merely performance on the four evaluation tasks. In §3.1 the knowledge base is built from 186,197 criminal and 152,452 civil legal documents, and the evaluation tasks are drawn from the same public Chinese legal corpora (CAIL2018 for Tasks #2 and #3, Xiezhi for Task #1, and laic.cjbdi.com for Task #4). The seed problems are constructed per task, and the generated data uses the exact answer formats of those tasks (e.g., Appendix B's '[金额]...<eoa>' format matches Task #4). If the knowledge base or DeepSeek V3's outputs contain the test documents or near-paraphrases of them, LAWGPT's fine-tuning memorizes test answers, invalidating the comparison to Lexilaw, ChatLaw, GPT-4, etc. The paper reports no overlap statistics or contamination analysis, and no real-data fine-tuning baseline is provided, so the observed gains could reflect distribution matching rather than legal reasoning ability. This is the most load-bearing concern because it threatens the validity of every headline comparison, and it is directly testable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KgDG, a knowledge-guided data generation framework for legal reasoning, with three components: KGGEN (knowledge-guided generation from a legal knowledge base and seed problems), KGFIX (reference and reasoning correction), and DAVER (verification and filtering), plus MITRA, a mixture training strategy that expands each verified problem into a direct-answer pair and a reasoning-path pair. Using DeepSeek V3 as the generator, the authors create 25K verified problems, expand them to 50K training examples, and fine-tune Qwen-2.5 models of 0.5B, 1.5B, and 3B parameters. The resulting LawGPT models are evaluated on four Chinese legal reasoning tasks (article prediction, two prison-term prediction tasks, and criminal damages calculation) and are reported to outperform existing legal-specific LLMs and to approach the performance of DeepSeek V3, with ablations and a data-scaling analysis.","tokens_in":10700,"tokens_out":4011,"duration_ms":36280,"significance":"If the central claim is correct, the paper makes a practical contribution: a 3B open-source model that outperforms 7B-33B legal-specific LLMs and comes close to a 671B proprietary-scale model would be a useful and economical result for legal NLP. The framework is clearly described, the code is promised to be public, the experiments span multiple model sizes, and the ablation and scalability studies are appropriate. However, the significance of the claim depends on two currently unresolved issues: the possibility of training/evaluation contamination, since the knowledge base and the test tasks draw on the same public legal corpora, and the absence of independent or human verification of the generated data, since DAVER and the generator are both DeepSeek V3.","major_comments":[{"comment":"The reported gains are vulnerable to training/evaluation contamination. The legal knowledge base is built from 186,197 criminal and 152,452 civil legal documents, and the evaluation tasks are drawn from the same public Chinese legal corpora (CAIL2018 for Tasks #2 and #3, Xiezhi for Task #1, and laic.cjbdi.com for Task #4). The synthetic training data is generated from documents sampled from this knowledge base and uses the answer formats of the test tasks (e.g., the '[金额]...<eoa>' format for Task #4). The paper reports no overlap statistics, no deduplication of generated training items against test instances or source documents, and no retraining experiment on a contamination-free subset. Because every headline comparison in Table 2 rests on these four tasks, this issue is load-bearing and directly testable. Please add a contamination analysis with overlap statistics and retrain LawGPT on a version of the 50K set from which any near-duplicate of the test instances or their source documents has been removed.","section":"§3.1, Table 2"},{"comment":"The claim that the 50K examples are 'high-quality' is not supported by independent evidence. DAVER uses DeepSeek V3, the same model that generated the data, to judge correctness, and the Reference Modifier and Reasoning Corrector are also DeepSeek V3 prompts; the knowledge base itself is not audited. This is a circular quality-control procedure: it demonstrates that one LLM accepts the outputs of another LLM, not that the legal content is correct. Please report a human expert audit on a random sample of the generated data (including inter-annotator agreement and an error taxonomy), or substantiate quality with an independent verifier, and report the DAVER pass rate together with examples of accepted and rejected items.","section":"§2.2, Appendix A"},{"comment":"The effectiveness of KgDG is not isolated against a conventional fine-tuning baseline. The comparisons in Table 1 are against the untrained Qwen-2.5 base models, so the gains could be attributed to any additional supervised fine-tuning data rather than to knowledge-guided generation. The ablation in Table 3 does include a KGGEN-only condition, but it does not include a condition that generates the same amount of data without the knowledge base and with the same correction/verification pipeline; the first row is again the untrained base model. To support the central claim that KgDG itself is effective, please add a baseline trained on an equal-sized dataset generated by the same pipeline without the legal knowledge base, and ideally a baseline trained on real legal data of the same size.","section":"§3.1, Table 1 and Table 3"},{"comment":"The conclusion that LawGPT improves 'legal reasoning' is broader than what the evaluation supports. The four tasks are all on Chinese criminal and civil law, and three of the four are derived from the same datasets used to construct the knowledge base. To support generalization beyond these four tasks, please either temper the wording or add out-of-domain evaluation tasks, such as legal QA from a different jurisdiction, contract or statute QA, or additional LawBench categories that were not used during data generation.","section":"§3.2, Table 2"}],"minor_comments":[{"comment":"The ablation study is reported on a single 4K subset without variance across random seeds or data subsets; please add standard deviations or multiple repeats, especially because the paper notes DAVER causes a slight degradation on Tasks #2 and #3.","section":"§3.3, Table 3"},{"comment":"The text says the Reference Modifier validates and corrects references 'using LLMs or the knowledge base,' but the Appendix A prompt for the Reference Modifier shows only an LLM prompt operating on the provided JSON; please clarify how the knowledge base is actually used in this step.","section":"§2.2 and Appendix A"},{"comment":"The y-axes of the four scalability plots use narrow ranges (e.g., 32.2-38.7 for Task #1 and 84.2-89.2 for Task #2), which makes small improvements appear larger than they are; please use full-scale axes and, where possible, add error bars.","section":"Figure 2"},{"comment":"There are several typos and grammatical issues: 'exising' in Section 2, 'evluation' in Section 3.1, 'seperated' in Section 2.3, 'out fine-tuned model' in Section 3.2, and 'Our code and resources is publicly available' in the abstract. A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study of a practical data-generation recipe, and the main concerns are testable rather than fatal. In my view the contamination issue in particular must be resolved before the comparisons to DeepSeek V3 and legal-specific LLMs can be considered trustworthy; a human audit or independent verifier is also needed for the data-quality claim. If the authors can supply a deduplication/contamination analysis, a retrained model on the cleaned data, and a human audit sample, the paper could become a useful contribution to the legal LLM literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a believable pipeline paper, and the ablation and scaling evidence are real, but the headline comparison has a load-bearing contamination risk that the paper does not address. I would send it to review, but only with a demand for dedup/overlap analysis and a real-data fine-tuning baseline.\n\nWhat is new: KgDG is the first application I know of that takes the MetaMath/WizardMath recipe—generate data with a strong model, refine, verify—and makes it work for Chinese legal reasoning by sampling from a legal knowledge base, fixing references and reasoning with a second pass, and verifying with a third. MITRA's mix of direct-answer and reasoning-path training instances is simple and sensible. The experiments are honestly presented: gains grow with data volume and model scale, and the ablation shows each module contributes, with DAVER slightly hurting on prison term prediction. That last detail is the kind of thing that makes me trust the paper is not cherry-picking.\n\nWhere it is soft: the contamination concern is real. The knowledge base is built from 186K criminal and 152K civil documents, and three of the four eval tasks come from public corpora like CAIL2018 and Xiezhi. The paper reports no overlap statistics between the KB, the generated 50K, and the test sets. If CAIL2018 test documents or near-paraphrases ended up in the KB, DeepSeek V3 could reproduce them in the synthetic data, and the fine-tuned model would be memorizing rather than reasoning. The seed problems use the exact answer formats of the eval tasks, which amplifies the risk. This is directly testable with n-gram overlap and a retrain on deduplicated data. They need to run it.\n\nSecond, DAVER uses DeepSeek V3 to judge data produced by DeepSeek V3. That is circular for the \"high-quality 50K\" claim. Some independent check—a sample of human legal review, or at minimum a different model as verifier—is needed. Third, there is no baseline where the same Qwen models are fine-tuned on real legal QA data of similar size. Without it, we do not know whether LawGPT's gains come from the KgDG pipeline or simply from more legal-format training data.\n\nNone of these are fatal to the method's plausibility. The framework is coherent, the writing is clear, and the limitations section is honest. The citation pattern is fine, building on WizardMath and the authors' own math data generation work. But the current paper overstates what it establishes: \"comparable to proprietary LLMs\" is not supported until leakage is ruled out.\n\nWho should read it: people building domain-specific synthetic data pipelines, especially in legal NLP. It deserves a serious referee, but the referee should insist on the contamination and baseline experiments. I would not cite the headline numbers in my own work yet.","headline":"Solid pipeline paper with real ablation evidence, but the headline comparison needs a contamination check and a real-data baseline before its numbers can be trusted.","tokens_in":11240,"tokens_out":3132,"would_cite":false,"duration_ms":27941,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 3B open-source model, LawGPT, fine-tuned on 50K knowledge-guided synthetic legal examples, outperforms legal-specific LLMs up to 33B and approaches proprietary LLMs on four Chinese legal reasoning tasks.","keywords":["legal reasoning","large language models","synthetic data generation","knowledge-guided generation","legal LLM","domain adaptation","fine-tuning","Chinese legal tasks"],"falsifier":"Have independent legal professionals check a random sample of, say, 500 examples from the released 50K dataset to see whether each cited law exists, each citation matches its stated content, and each legal and arithmetic conclusion is correct; any substantive share of false citations among DAVER-verified examples would falsify the data-quality claim.","tokens_in":10294,"feed_emoji":"⚖️","tokens_out":8248,"duration_ms":66483,"temperature":0.7,"pith_summary":"This paper tries to establish that a small open-source model can reach near-proprietary legal reasoning ability if it is trained on synthetic data generated with legal knowledge. It proposes KgDG, a pipeline that samples real legal documents, writes new question-answer pairs with reasoning and references, repairs wrong references and flawed reasoning, filters unverifiable drafts, and trains the model on both direct and step-by-step answers. Using this pipeline on 50K generated examples, the authors report that LawGPT, a 3B fine-tuned model, outperforms legal-specific LLMs of 7B to 33B and scores close to GPT-4 and DeepSeek V3 on four Chinese legal reasoning tasks. If the result holds, it matters because legal AI could be built on open weights, with lower inference cost and fewer data-privacy concerns than calling a proprietary API for every query.","feed_headline":"3B LawGPT beats 7B-33B legal LLMs, matches GPT-4","feed_subtitle":"50K synthetic legal examples let a 3B open model rival GPT-4 and DeepSeek V3 on four Chinese tasks.","key_machinery":"The mechanism carrying the argument is the KgDG pipeline (Knowledge-Guided Data Generation). It couples a Knowledge-Aware Sampler, which selects legal documents and seed problems to keep synthetic questions diverse and grounded, with a Knowledge-Guided Writer that drafts question-answer pairs along with legal references and reasoning paths. The Knowledge-Guide Fixer then corrects wrong statute citations and faulty reasoning, and the Data Verifier filters out examples whose answers cannot be derived from the corrected references and reasoning. Mixture Training completes the mechanism by formatting each verified example twice, once as a direct answer and once with explicit reasoning, so fine-tuning teaches both efficient answering and step-by-step legal computation.","core_discovery":"The central claim is that knowledge-guided synthetic data can substitute for scarce, expensive human legal annotations. Concretely, the paper constructs a knowledge base of 186,197 criminal and 152,452 civil legal documents, samples documents together with ten seed problems per task, and asks DeepSeek V3 to write new questions, answers, reasoning paths, and legal references. A Reference Modifier and Reasoning Corrector patch the drafts, and a Data Verifier rejects drafts where the answer does not follow from the corrected reasoning and references. The 25K verified problems are expanded by Mixture Training into 50K training examples, then used to fine-tune Qwen-2.5 at 0.5B, 1.5B, and 3B. On the four tasks, LawGPT-3B averages 71.8, ahead of all legal-specific baselines (best 56.1) and GPT-4 (67.4), while DeepSeek V3 averages 74.2; ablations show the best average only when all four pipeline modules are active.","pith_inferences":["The paper leaves the generator and verifier as the same model family, so it does not isolate how much of DAVER's filtering is genuine verification versus stylistic self-consistency; a human-annotated audit of the released dataset would test this.","A natural extension is to replay KgDG with a different generator or with an external legal-validity checker; if performance holds, the framework's contribution is the knowledge-sampling structure rather than any particular generator.","Because the four tasks are all template-heavy Chinese legal reasoning problems, the method's transfer to open-ended legal advice or common-law jurisdictions is untested."],"forward_implications":["If the result holds, a 3B open-source model can deliver legal-reasoning quality close to GPT-4 and DeepSeek V3 on the four tested Chinese legal tasks, which makes private and low-cost legal inference feasible.","Because performance rose monotonically as training data grew from 8K to 50K, the framework appears to have headroom for larger synthetic datasets.","The MITRA expansion implies that training on both direct answers and explicit reasoning paths can improve accuracy without sacrificing response format.","Legal-specific models at 7B-33B are outperformed by a 3B model trained on synthetic data, suggesting scale is not the only route to domain competence."],"supporting_citations":[{"why":"Defines LawBench and the four Chinese legal reasoning tasks and metrics (ROUGE-L, normalized log-distance, accuracy) used for all model comparisons.","marker":"Fei et al., 2023"},{"why":"Serves as the LLM that generates draft legal data, fixes references and reasoning, and verifies correctness inside KgDG.","marker":"DeepSeek-AI et al., 2024"},{"why":"Supplies the Qwen-2.5 0.5B/1.5B/3B checkpoints that are fine-tuned into LawGPT.","marker":"Yang et al., 2024"},{"why":"Provides the LLaMA-Factory fine-tuning harness used to train LawGPT on the 50K synthetic examples.","marker":"Zheng et al., 2024"},{"why":"Establishes the prior approach of LLM-based synthetic data generation for math reasoning that KgDG adapts to legal reasoning with knowledge guidance and verification.","marker":"Li et al., 2024b"},{"why":"Underlies the prison-term prediction tasks (#2 and #3) whose data and labels are used to evaluate LawGPT's sentencing predictions.","marker":"Xiao et al., 2018"}],"fun_headline_variants":["3B LawGPT beats 7B-33B legal LLMs, matches GPT-4 via synthetic data","Knowledge-guided synthetic data lets 3B model match GPT-4 on legal tasks","50K guided synthetic examples make 3B legal LLM rival GPT-4 and DeepSeek V3","LawGPT: synthetic legal data from DeepSeek V3 lifts 3B model to GPT-4 parity","Tiny 3B model equals GPT-4 in legal reasoning with knowledge-guided data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline's quality guarantee rests on the assumption that a legal-knowledge base plus the same generative LLM's self-correction and verification can catch every legally significant hallucination, so that the 50K synthetic examples are genuinely correct.","fun_headline_variants_meta":{"raw":{"variants":["3B LawGPT beats 7B-33B legal LLMs, matches GPT-4 via synthetic data","Knowledge-guided synthetic data lets 3B model match GPT-4 on legal tasks","50K guided synthetic examples make 3B legal LLM rival GPT-4 and DeepSeek V3","LawGPT: synthetic legal data from DeepSeek V3 lifts 3B model to GPT-4 parity","Tiny 3B model equals GPT-4 in legal reasoning with knowledge-guided data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1816,"prompt_tokens":995,"completion_tokens":821,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":696}},"tokens_in":611,"tokens_out":821,"duration_ms":7913,"temperature":1.0,"reasoning_tokens":696,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:00:18.927614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent legal professionals check a random sample of, say, 500 examples from the released 50K dataset to see whether each cited law exists, each citation matches its stated content, and each legal and arithmetic conclusion is correct; any substantive share of false citations among DAVER-verified examples would falsify the data-quality claim.","supporting_citations":[],"review_version":1}