{"id":"b787d9c5-397c-43e4-8bd0-89b5530311a8","arxiv_id":"2507.06539","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Clustering contracts, LLM-based labeling, augmentation, and LoRA fine-tuning improve Chinese industrial contract field extraction over traditional TF-IDF/TextRank/SNOWNLP/KeyBERT baselines.","lead":"This paper builds a Chinese industrial contract dataset by having GPT-4 and GPT-3.5 label clustered contracts, augments it with synthetic texts, and fine-tunes open LLMs with LoRA. The fine-tuned Qwen2.5 model extracts contract fields with higher ROUGE scores than traditional keyword methods, and the authors report that data balancing and augmentation help.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported accuracy and ROUGE figures rest on unverified GPT-3.5-generated labels; if the test labels come from the same pipeline, the numbers measure agreement with the teacher model rather than true contract content.","rationale":"I read the paper in good faith as an applied NLP system paper whose contribution is a pipeline for constructing a training set and fine-tuning an LLM for Chinese industrial contract extraction. The pipeline is coherent and the inclusion of weak traditional baselines plus a small ablation with standard deviations is useful. However, the single most load-bearing assumption is the trustworthiness of the labels used for both training and evaluation. The paper says in Section 3.2 that manual review was applied to \"a portion\" of the annotations, but gives no evidence of its extent or outcome, and it never describes how the test-set gold labels were obtained. If the test labels are official human-verified annotations from the Obei platform, the circularity concern is largely resolved; if they are generated by the same GPT-3.5/GPT-4 process, the headline numbers are self-consistency scores. Because this distinction completely changes what the experiments demonstrate, it is more fundamental than secondary issues such as missing confidence intervals or hyperparameters selected on the test set. The proposed concrete check, human annotation of a test subset with agreement statistics, would settle the concern directly. I therefore agree with the reader's weakest-assumption analysis and recommend keeping the conditional verdict: accept only if the label-quality evidence is supplied or the claims are reframed.","tokens_in":10284,"tokens_out":4995,"duration_ms":58013,"concrete_test":"Independently recruit two Chinese-speaking annotators, blinded to model outputs, to label a stratified random sample of 50-100 test contracts using the same field schema. Compute inter-annotator agreement and compare (a) the GPT-3.5/GPT-4 labels and (b) Qwen2.5-Final outputs against the human majority labels using the paper's exact metrics (Eq. 4-7, edit distance <= 3). Also report the size and correction rate of the manual review in Section 3.2 Step 3. If Qwen2.5-Final's human-label scores are statistically indistinguishable from the reported values, the concern is resolved; if they drop materially, the headline should be reframed as measuring agreement with LLM-generated labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that the fine-tuned model achieves high field-level accuracy and recall, is anchored by the metrics in Section 4.2, which compare model outputs against \"gold-standard structured text.\" The paper never states how the test-set labels were produced. The only labeling description is Section 3.2, where GPT-4 creates one exemplar per cluster and GPT-3.5 batch-annotates the remaining texts via Eq. (3); Step 3 says only that \"a portion\" of the annotations is manually reviewed, with no sample size, no correction rate, and no inter-annotator agreement. If the evaluation labels are generated by the same GPT-3.5/GPT-4 procedure, then fine-tuning Qwen2.5 on those labels and evaluating against them does not establish extraction correctness; it establishes consistency with an unvalidated automatic annotator. ROUGE is particularly sensitive to surface-form overlap, so a model trained to imitate GPT-3.5's output format could score well even when field values are wrong. The reported total field accuracy of 0.4724 and ROUGE-1 of 0.6296 therefore may be inflated by label self-consistency. This concern is load-bearing because it determines whether the quantitative anchor of the paper measures real contract information or LLM agreement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a pipeline for constructing a labeled Chinese industrial contract information extraction dataset: TF-IDF/K-means clustering, per-cluster GPT-4 exemplar annotation, GPT-3.5 batch annotation via in-context learning, a manual review step whose details are not quantified, and keyword-recombination-based data augmentation. The resulting dataset is used to LoRA fine-tune three open-source LLMs (GLM4-9B, Baichuan2-13B, Qwen2.5-14B) and evaluate them on the Obei test set. The best model, Qwen2.5-Final, reaches a total field accuracy of 0.4724, ROUGE-1 of 0.6296, ROUGE-2 of 0.5008, ROUGE-L of 0.5791, and 0.48 contracts/sec. The paper concludes that the method achieves high field-level precision and recall while maintaining efficient parsing.","tokens_in":10557,"tokens_out":7586,"duration_ms":73376,"significance":"If the reported evaluation is valid, the proposed pipeline would be a practical, low-cost approach for domain-specific information extraction data construction, and the comparison of three open-source LLMs with LoRA fine-tuning is useful for practitioners. The ideas of cluster-conditional exemplars for in-context annotation and keyword-recombination augmentation to decouple surface patterns are interesting and potentially transferable. However, the current evidence is insufficient to support these strengths: the gold-standard labels are produced by the same automatic annotator used for training, the manual quality check is unquantified, and the central quantitative claims rest on this circular evaluation. The paper's contribution would be much more compelling with independent test labels or a documented, large human-validated subset.","major_comments":[{"comment":"The paper never establishes that the test-set gold standard is independent of the automatic annotator. Step 3 of Section 3.2 states only that 'a portion of the annotation results in Yi is sampled for manual review to ensure quality,' without reporting the sample size, the correction rate, or inter-annotator agreement. Section 4.2 evaluates against 'gold-standard structured text' but does not state how the test labels were produced. Since the training labels come from GPT-3.5 using the Eq. (3) in-context procedure, and the test set appears to be labeled by the same pipeline, the field accuracy and ROUGE scores in Table 2 may measure the fine-tuned model's agreement with the GPT-3.5 teacher rather than correctness on real contract content. The authors should use the official Obei test labels if these exist, or report a rigorous manual validation (e.g., a random sample of a few hundred test instances with human-corrected labels and the resulting error rate).","section":"Section 3.2 (Step 3) and Section 4.2"},{"comment":"The pseudocode for the WCSS-based cluster count selection is incorrect and irreproducible. It computes the squared distance of each cluster (treated as a single vector) to the global mean of all data points, not the within-cluster sum of squared distances from individual data points to their cluster centroids as defined in Eq. (2). The selection condition 'if bestwcss >= WCSS' stores a smaller WCSS, but the text says the 'highest clustering score' is chosen; since WCSS is a cost to minimize, this would trivially select the largest k tested. The reported choice of 9 clusters for qualification texts and 10 for requirement texts in Section 4.1 therefore cannot be reproduced from the description. The authors should correct the pseudocode and explicitly state the actual cluster-selection criterion.","section":"Algorithm 1"},{"comment":"The efficiency metric (contracts per second) is reported without the measurement conditions needed for reproducibility. The paper does not specify the prompt length, maximum number of generated tokens, batch size, decoding strategy, whether GPU warmup was performed, or whether timing was averaged over multiple runs. Since 'efficient parsing' is part of the claimed contribution, the efficiency numbers (0.48-0.81 contracts/sec in Table 2) should be accompanied by a description of the inference setup and hardware (Section 4.3 specifies the training GPU only).","section":"Section 4.2 and Table 2"}],"minor_comments":[{"comment":"There is an inconsistency about which LLM performs the augmentation: Section 3.3 Step 4 says 'prompt a large language model' while earlier text refers to GPT-3.5, and Section 5 says GPT-3.5 annotates randomly recombined keywords, which conflates annotation with text generation; please clarify.","section":"Section 3.3 / Section 5"},{"comment":"Table 6 reports only the mean and standard deviation of ROUGE-1 for the four ablation conditions, yet the text claims effects on 'prediction accuracy' and 'model robustness'; the authors should also report field accuracy and at least ROUGE-2/ROUGE-L for these conditions.","section":"Table 6"},{"comment":"The paper does not describe the target schema (the set of fields to be extracted for qualification and requirement texts), so the reader cannot assess what the field accuracy and ROUGE scores actually measure; a list or example of the target keys should be provided.","section":"Section 4.1"},{"comment":"The statement that the large language models 'demonstrate extraction performance that meets real-world business requirements' is not justified by any quantitative threshold or user evaluation; please either provide such a benchmark or soften the claim.","section":"Section 4.4"},{"comment":"The sizes of the stratified subsets C'_i and the exact number of augmented samples (10% of what?) are not reported, making the training-set composition unclear.","section":"Section 4.1 / Section 3.2"},{"comment":"The two workflow figures are referenced in the text but are not legible in the submitted PDF; they should be replaced with readable versions that show the actual prompts and extracted outputs.","section":"Figures 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The label-provenance issue is the main barrier to acceptance. In the current form, the quantitative anchor could be entirely an artifact of teacher-student self-consistency, and the paper does not rule this out. If the authors cannot access official Obei labels or produce a human-verified test subset, I would lean toward rejection despite the interesting pipeline ideas. The Algorithm 1 pseudocode error also suggests the manuscript needs a careful technical proofreading pass before it can be considered reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent applied pipeline paper for Chinese industrial contract extraction, with a sensible clustering-plus-LLM-annotation design and a useful keyword-recombination augmentation. The soft spot is that the evaluation labels appear to be generated by the same GPT-3.5/GPT-4 pipeline, and the manual review is described as 'a portion' with no numbers. If that's true, the headline accuracies and ROUGE scores measure agreement with the teacher model, not extraction correctness.\n\nThe actual new content is the combination of TF-IDF/K-means clustering with one GPT-4 exemplar per cluster used as in-context demonstration for GPT-3.5 batch annotation, plus the augmentation trick of recombining keywords and having GPT-3.5 write synthetic contracts. Both components are known individually, but the specific integration is clean and the ablation study (balanced vs imbalanced, with/without augmentation, with standard deviations over three runs) is a nice touch. The efficiency numbers are also practical for deployment.\n\nThe biggest problem is label validation. The paper never states how the test set labels were produced. Section 3.2, Step 3, says only that 'a portion' of annotations is manually reviewed, with no sample size, inter-annotator agreement, or correction rate. If the test labels come from the same GPT-3.5 batch annotation, then fine-tuning on those labels and evaluating against them is partly circular. ROUGE makes this worse because it rewards surface-form overlap, so a model trained to imitate GPT-3.5's output format can score well even when field values are wrong. The reported 0.4724 field accuracy and 0.6296 ROUGE-1 therefore can't be taken at face value. That's a load-bearing issue because the main claim is that the model extracts information accurately.\n\nOther soft spots are smaller: the traditional baselines (TF-IDF+UIE, TextRank, etc.) are so weak that beating them is not meaningful; hyperparameters (LoRA rank, epochs) are selected on the test set; the test set is only 174 contracts; and no code or data are released.\n\nNone of this makes the pipeline worthless. As a recipe for building a contract extraction model with limited manual annotation, it's plausible and well described. But the evidence in this version doesn't support the accuracy claims. I'd send it to review with a request for label-quality statistics, independent test labels or human verification, error bars on the main table, and artifacts. An applied NLP group working on document extraction would find the method worth trying, but they should validate the labels themselves.\n\nMy recommendation: press for revision rather than desk reject, but don't accept the numbers as-is.","headline":"A practical contract-extraction pipeline whose headline numbers rest on unverified LLM-generated labels, so they may measure self-consistency rather than accuracy.","tokens_in":11085,"tokens_out":3223,"would_cite":false,"duration_ms":72670,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a fully automated pipeline—clustering with TF-IDF and K-means, GPT-4 exemplar annotation, GPT-3.5 batch annotation, keyword-recombination augmentation, and LoRA fine-tuning—produces a model that extracts structured…","keywords":["contract information extraction","large language models","data annotation","data augmentation","LoRA fine-tuning","TF-IDF clustering","Chinese industrial contracts","ROUGE evaluation"],"falsifier":"Take a random sample of test contracts, have two human experts independently extract the fields using the same edit-distance tolerance, and measure agreement with GPT-3.5's labels; if the human-to-GPT agreement is close to the reported field accuracy, the model is matching the annotator rather than the contract content.","tokens_in":10068,"feed_emoji":"📄","tokens_out":7044,"duration_ms":65077,"temperature":0.7,"pith_summary":"The paper aims to show that high-quality labeled data for extracting structured fields from complex Chinese industrial contracts can be produced without large-scale human annotation, and that a locally deployed open-source model fine-tuned on that data can outperform traditional keyword-extraction pipelines while staying fast enough for practical use. The authors argue that clustering raw contracts, letting GPT-4 create one example per cluster, and having GPT-3.5 annotate the rest yields a training set whose quality is adequate for fine-tuning; adding synthetic contracts generated from randomly recombined keywords and balancing clusters further improves accuracy and consistency. If the claim holds, contract-heavy organizations could replace slow manual review with an automated system that keeps most of the accuracy benefits of a large model at a fraction of the labeling cost.","feed_headline":"GPT-assisted pipeline lifts contract extraction to 0.47 accuracy","feed_subtitle":"Clustering, GPT-4 exemplars, and keyword recombination beat keyword baselines while parsing 0.48 contracts per second.","key_machinery":"The core mechanism is a cluster-then-annotate loop: TF-IDF encoding plus K-means partitions raw contracts into clusters, GPT-4 writes one gold exemplar per cluster from the sample closest to each cluster center, and GPT-3.5 uses that exemplar in the prompt to annotate all remaining texts in the cluster. A keyword-recombination augmentation step then extracts keywords from annotations, randomly combines 5–6 keywords, and asks GPT-3.5 to generate new unstructured contract texts, which breaks the model's reliance on fixed keyword patterns. LoRA fine-tuning of an open-source Chinese LLM on the resulting balanced and augmented dataset carries the argument from data construction to deployed extraction performance.","core_discovery":"On the paper's own terms, the central discovery is that a fully automated annotation pipeline—TF-IDF/K-means clustering, one GPT-4 exemplar per cluster, GPT-3.5 batch annotation using that exemplar, keyword-recombination augmentation, and LoRA fine-tuning—produces a model, Qwen2.5-Final, that reaches 0.4724 total field accuracy, ROUGE-1/2/L scores of 0.6296/0.5008/0.5791, and 0.48 contracts per second on the Obei test set. Each pipeline component contributes measurably: LoRA fine-tuning improves ROUGE-1 over the base model, data balancing raises mean ROUGE-1, and data augmentation raises it further while lowering the standard deviation across repeated training runs. The paper concludes that the approach meets real-world business requirements while remaining efficient enough for batch processing.","pith_inferences":["Editorial inference: because the reported metrics compare against labels produced by GPT-3.5, the results may primarily demonstrate that the model reproduces GPT-3.5's annotation behavior; an independent human-labeled test set could show whether real contract content is being captured.","Editorial inference: the same cluster-then-annotate plus keyword-recombination recipe could transfer to other semi-structured documents, such as invoices, permits, or medical records, without rewriting the pipeline, provided a large language model can produce reliable exemplars in that domain.","Editorial inference: the augmentation step adds only 10% synthetic data, so scaling the fraction of generated contracts and varying the keyword count may reveal a trade-off between robustness and fidelity to real contract language.","Editorial inference: if independent labels confirm the approach, a testable extension is to replace GPT-3.5 with a smaller open-source annotator, removing the dependence on proprietary APIs from the data-creation loop."],"forward_implications":["Training a contract-extraction model no longer requires human annotation of every document: one GPT-4 exemplar per cluster plus GPT-3.5 batch annotation is enough to fine-tune an open-source model.","LoRA fine-tuning of a 14B-parameter model on a single 48GB GPU produces larger gains for Qwen2.5 than for GLM4 or Baichuan2, with ROUGE-1 improving by 41.55% in that configuration.","Balancing the cluster-imbalanced training data raises mean ROUGE-1, and keyword-recombination augmentation raises it further while lowering standard deviation across repeated runs.","The final model parses contracts at 0.48 contracts per second, making the approach deployable in batch industrial settings rather than only as an offline analysis tool.","The pipeline beats TF-IDF+UIE, TextRank+UIE, SnowNLP+UIE, and KeyBERT+UIE on every reported metric."],"supporting_citations":[{"why":"Supplies the Intelligent Contract Information Extraction Dataset of Chinese industrial contracts with training and test splits used in all experiments.","marker":"Obei, 2023"},{"why":"Provides the Qwen technical report and the Qwen2.5 base model that becomes Qwen2.5-Final after fine-tuning.","marker":"Bai et al., 2023"},{"why":"Provides the GLM pretraining method corresponding to GLM4-9B, one of the three compared base models.","marker":"Du et al., 2021"},{"why":"Introduces Baichuan2-13B-Chat, the second compared base model used before and after fine-tuning.","marker":"Yang et al., 2023"},{"why":"Defines the ROUGE-1, ROUGE-2, and ROUGE-L metrics used to approximate field recall.","marker":"Lin, 2004"},{"why":"Provides the inverse document frequency justification for the TF-IDF representation used in clustering.","marker":"Robertson, 2004"},{"why":"Supplies the UIE framework that combines keyword-extraction baselines into structured information extraction comparisons.","marker":"Lu et al., 2022"}],"fun_headline_variants":["Clustered GPT annotations train contract extractor to 0.47","Automated GPT pipeline yields efficient contract parsing at 0.48/s","LoRA and augmentation boost contract extraction to 0.47 accuracy","Industrial contract extraction hits 0.47 via GPT-guided pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on GPT-3.5's labels being correct: only a portion of them is manually reviewed and the test labels are never independently verified.","fun_headline_variants_meta":{"raw":{"variants":["Clustered GPT annotations train contract extractor to 0.47","Automated GPT pipeline yields efficient contract parsing at 0.48/s","LoRA and augmentation boost contract extraction to 0.47 accuracy","Industrial contract extraction hits 0.47 via GPT-guided pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000421,"raw_usage":{"total_tokens":2132,"prompt_tokens":877,"completion_tokens":1255,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":1179}},"tokens_in":493,"tokens_out":1255,"duration_ms":10193,"temperature":1.0,"reasoning_tokens":1179,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:59:26.708690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of test contracts, have two human experts independently extract the fields using the same edit-distance tolerance, and measure agreement with GPT-3.5's labels; if the human-to-GPT agreement is close to the reported field accuracy, the model is matching the annotator rather than the contract content.","supporting_citations":[],"review_version":1}