{"id":"196c9a44-ae0a-483c-9641-8db089826625","arxiv_id":"2412.19482","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A three-stage framework combines legal-domain pre-training, circle-loss fine-tuning, and similarity-aggregation re-ranking to improve Chinese legal question answering.","lead":"PFR-LQA is a three-stage pipeline, domain pre-training, fine-tuning, and contextual re-ranking, for retrieving legal question-answer pairs. It reports gains of about 6 points in P@1 over the strongest baseline on a new 549k-pair Chinese legal QA dataset, though the test set is small and no code or data is released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LawQA's 900 test pairs may not be 900 independent units: the split unit is unspecified, and duplicate queries or train-test overlap could alone explain the 5.7-point P@1 gap. The empirical claim needs a query-level split and leakage check before it can be accepted.","rationale":"The central claim is empirical: PFR-LQA beats SimCSE by 5.7 points P@1 and 4.2 points MRR@16 on LawQA. For that claim to hold, the evaluation must use independent, representative test instances and a clean separation from training. The paper is otherwise coherent: the three stages are described in enough detail to reproduce the training objectives, the ablations in Table II have the expected ordering (task-specific fine-tuning matters most), and the dataset is large. The weakest point is the black-box construction of LawQA: no data release, no split description, no significance tests. The reader identified the need for a representative test set and error bars; I sharpen that into a concrete leakage and independence risk. Because the paper reports 549,668 positive QA pairs and Table I lists 322,475 cases, many queries must have multiple gold answers. A random split of QA pairs will place the same query in multiple splits, violating the independence assumption of any paired test and creating a shortcut for the BM25-based re-ranking stage. With only 900 test pairs, the 5.7-point gap is about 51 questions; if those concentrate on a few leaked or duplicated queries, the SOTA claim collapses. The proposed check is decisive: query-level aggregation and train/test deduplication tell us whether the gap is real. If the gap survives both, the reader's conditional acceptance can be upgraded; if not, the paper's central claim is an artifact of data construction rather than of retrieval quality. Until then, keeping the conditional verdict is appropriate.","tokens_in":7767,"tokens_out":13892,"duration_ms":132344,"concrete_test":"Count distinct user queries among the 900 test QA pairs; if duplicates exist, recompute P@1 and MRR at query level and pair-bootstrap PFR-LQA against SimCSE with 10,000 resamples. Separately, detect train/test overlap by exact match and BM25 top-5 similarity against the 549,668 training positives, excluding the test gold; rerun Table II on the deduplicated subset. If the 5.7-point P@1 gap becomes statistically insignificant or reverses in either analysis, the headline advantage is an artifact of the split rather than a property of the three-stage framework.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A specifies only the number of training/validation/test pairs and never the split unit. LawQA is built from forum user queries with lawyer-written answers; the 322,475 'cases' in Table I vs. 549,668 positive QA pairs imply that a single query can have multiple gold answers. If the 900 test pairs were sampled at the QA-pair level rather than the user-query level, the same query can appear in multiple test pairs and in the training fold. The reported P@1 and MRR are then not computed over 900 independent units, and a P@1 gap of 5.7 points (~51 pairs) could be driven by a handful of duplicated or leaked queries. The re-ranking stage makes this worse: it deliberately retrieves the query's nearest questions by BM25 and re-ranks by question affinity, so an exact or near-duplicate training question is a shortcut to the gold answer. No significance test, confidence interval, or train/test overlap analysis is reported, so the Table II advantage over SimCSE is currently indistinguishable from a test-construction artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PFR-LQA, a three-stage framework for Chinese legal question answering. Stage one pre-trains a dual-encoder model (Legal-SCP) on legal questions and answers with self-supervised and context-supervised masked auto-encoding objectives; stage two fine-tunes the encoder on question-answer pairs with circle loss; stage three re-ranks candidates by computing affinity features against anchors and refining them with a Transformer, using a contrastive loss plus an MSE reconstruction loss. The authors construct a large Chinese legal QA dataset called LawQA with 549,668 positive QA pairs and report experiments comparing PFR-LQA with BM25, BERT, RoBERTa, LawFormer, DPR, ColBERT, and SimCSE. On a 900-pair test set, the proposed method achieves P@1 of 79.9 and MRR@16 of 87.3, outperforming the best baseline SimCSE by 5.7 and 4.2 points, respectively. Three ablation variants (w/o DSP, w/o TSF, w/o CR) are included in Table II.","tokens_in":7994,"tokens_out":4287,"duration_ms":43850,"significance":"If the reported results are reliable, PFR-LQA would demonstrate that combining domain-specific pre-training, task-specific dense retrieval fine-tuning, and contextual re-ranking yields a meaningful advance for community-style legal question answering, a practically important task. The construction of a large legal QA corpus with lawyer-written answers is also a potentially useful resource for the community, provided the dataset is released with a clear and leakage-safe evaluation protocol. The paper's high-level idea is reasonable and builds on established components (Legal-SCP, circle loss, and similarity aggregation), but the central empirical claim is currently not supported at the level of rigor expected for a state-of-the-art claim: the evaluation lacks statistical validation, the test set construction is underspecified, and the re-ranking stage contains an underspecified threshold-based labeling procedure.","major_comments":[{"comment":"The construction and sampling of the 900-pair test set are not described, and the split unit is never specified. Table I reports category sizes in terms of 'cases' (322,475 total), while the text reports 549,668 positive QA pairs, which implies that a single case or user query can have multiple gold answers. If the 900 test pairs were sampled at the QA-pair level rather than at the query level, the same query can appear in both training and test folds, and the reported P@1 gap of 5.7 points corresponds to only about 51 test pairs. The authors should specify the split unit, perform a query-level split, report train/test overlap statistics, and provide confidence intervals or significance tests for the pairwise differences in Table II. Without this, the claimed advantage over SimCSE is indistinguishable from a test-construction artifact.","section":"III-A and III-D"},{"comment":"The positive/negative labeling used to train the contextual re-ranking stage is self-referential and is not fully specified. The paper states that 'question pairs with small similarities (less than a threshold)' are treated as negative samples and the remaining retrieved candidates as positive samples, but the threshold value is not given, and no justification is provided for treating all remaining candidates as positive. In addition, the procedure for selecting the L anchor questions from the K candidates is not described. These details are load-bearing because the re-ranking stage is one of the three claimed contributions, and the choice of threshold can determine whether the stage learns genuinely new information or merely re-ranks according to the fine-tuned model's own similarities.","section":"II-C and III-C"},{"comment":"No measure of variability is reported anywhere in the experiments. There are no standard deviations over multiple runs, no random seeds, and no significance tests for the differences in Table II or for the ablation rows. Given that the test set has only 900 pairs, a 5.7-point P@1 gap is about 51 questions, and the ablation gaps (e.g., 4.2 points between PFR-LQA and w/o DSP) are of similar magnitude. The authors should report confidence intervals, bootstrap estimates, or significance tests before claiming that the three-stage framework yields substantial improvements.","section":"III-D and III-E"},{"comment":"The ablation study is described in a single sentence and does not specify how each component is removed. For instance, 'w/o DSP' might mean starting from RoBERTa without legal-domain pre-training, and 'w/o CR' might mean returning the BM25 top-K candidates without re-ranking, but the text does not say. It is also unclear whether hyperparameters are re-tuned for each ablation. Since the paper's contribution is explicitly the collaborative effect of the three stages, the ablation must state what exactly is removed and how the remaining pipeline is retrained; otherwise the relative contribution of each stage cannot be assessed.","section":"III-E"}],"minor_comments":[{"comment":"The section numbering is inconsistent: 'III. EXPERIMENTS' is immediately followed by 'IV. EXPERIMENTAL SETUP', and the experimental content actually appears under the latter heading. The headings should be renumbered.","section":"Section headings"},{"comment":"Please clarify the relation between the 549,668 positive QA pairs and the 322,475 total 'cases' in Table I. The table caption and the surrounding text use different units, and the number of QA pairs per case is not stated.","section":"III-A"},{"comment":"The phrase 'randomly select several negative candidates via the BM25 algorithm' is vague. The number of negative candidates sampled per positive pair and the size of the BM25 candidate pool should be stated, especially because this sampling determines the training and validation set sizes.","section":"III-A"},{"comment":"The index range '1 ≤ i ≤ L ≤ K' appears to contain a typo, since the equation uses j for the candidate question and i does not appear elsewhere; it should likely be '1 ≤ j ≤ L ≤ K'.","section":"II-C, Eq. (4)"},{"comment":"There are several typos and stylistic issues, including 'fune-tuning' in Section IV-C, 'peform' in Section III-E, and 'we use the pre-trained Legal-SCP model' in Section II-C, which should be 'the fine-tuned Legal-SCP model' for clarity.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is an application-oriented combination of existing techniques (Legal-SCP pre-training, circle loss, and similarity aggregation for re-ranking) on a new legal QA dataset. The main uncertainty is the evaluation: the 900-pair test set, the unspecified split unit, and the absence of significance testing leave the state-of-the-art claim unsupported. I recommend major revision with a request for a leakage-safe evaluation protocol and full specification of the re-ranking training details. If the authors can release the dataset and code with a clear query-level split and statistical validation, the contribution could be solid for a venue interested in applied QA systems."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable engineering paper, and the LawQA dataset is the real asset. The three stages are all borrowed — CMAE-style pre-training, circle loss fine-tuning, and a similarity-aggregation re-ranker close to Chen et al. [4] — but the combination is coherent and the authors don't hide the debts. I'd credit them for building a 549k-pair lawyer-annotated dataset and for running ablations that show each stage contributes.\n\nThe soft spots are in the evaluation, and they're not minor. The test set is 900 pairs. The paper reports no significance tests, no confidence intervals, no multiple seeds. A 5.7-point P@1 gap over SimCSE is about 51 questions; that's within plausible noise for 900 items.\n\nThe deeper problem is that the split unit is never specified. LawQA has 322,475 queries but 549,668 positive pairs, so a single query has multiple gold answers. If the 900 test pairs were sampled at the QA-pair level rather than the query level, the same query can appear in both train and test. And with BM25 candidate retrieval plus a re-ranking stage that deliberately exploits question-question affinity, a near-duplicate training question is a direct shortcut to the gold answer. The authors need to state the split unit, do a query-level split, and report overlap/leakage analysis. Without that, the Table II advantage is hard to distinguish from a test-construction artifact.\n\nThere's also a mild self-referential element in the re-ranking training: positive/negative labels for candidate questions are assigned using the fine-tuned model's own similarity scores with an unspecified threshold. The threshold, the anchor selection, and the number of anchors are not detailed. And a small hit: HanLP is cited to a paper that is not about HanLP.\n\nOn the plus side, the central idea is plausible and the method is clearly described within each stage. The paper is not incoherent; it's just under-evidenced. If the dataset is actually released (the paper says 'the release of them' but gives no link), this becomes more valuable.\n\nWho's this for? People building dense retrieval for high-stakes domains, and anyone thinking about evaluation leakage in community QA. It deserves a real referee, but the referee should push for data release, query-level splitting, error bars, and a clear description of the re-ranking label construction. My vote: send to peer review, conditional acceptance.","headline":"A sensible three-stage legal QA pipeline with a potentially valuable dataset, but the evaluation rests on a 900-pair test set whose construction and split unit are unspecified, so the headline gains aren't yet trustworthy.","tokens_in":8502,"tokens_out":2909,"would_cite":false,"duration_ms":27285,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three-stage legal QA framework—domain pre-training, fine-tuning, and contextual re-ranking—beats the strongest baseline by 5.7 points P@1.","keywords":["legal question answering","dense retrieval","dual-encoder","domain-specific pre-training","contextual re-ranking","similarity aggregation","LawQA dataset","Chinese legal QA"],"falsifier":"Bootstrap-resample the 900 LawQA test pairs from the full positive-pair pool and compute a 95% confidence interval for the P@1 difference between PFR-LQA and SimCSE; if the interval contains zero, the claimed 5.7-point superiority is not established.","tokens_in":7572,"feed_emoji":"⚖️","tokens_out":8999,"duration_ms":75885,"temperature":0.7,"pith_summary":"The paper tries to establish that legal question answering is best served by a three-stage pipeline rather than a single dual-encoder fine-tuning step: domain-specific pre-training on legal text, task-specific fine-tuning on question-answer pairs, and a contextual re-ranking stage that refines query representations with similar questions as anchors. On a new lawyer-annotated Chinese dataset called LawQA, the full pipeline reports 79.9% precision at the first position (P@1) and 87.3 MRR@16, beating the strongest baseline, SimCSE, by 5.7 and 4.2 points. The paper also reports ablations in which removing fine-tuning hurts most, while removing domain pre-training or re-ranking each costs a smaller but consistent amount. If the result holds, legal QA systems can be improved without additional labeled data, by exploiting unlabeled legal corpora and the structure of the QA database itself.","feed_headline":"Three-stage legal QA pipeline tops SimCSE by 5.7 points","feed_subtitle":"Domain pre-training, fine-tuning, and contextual re-ranking lift precision on 900 LawQA test cases.","key_machinery":"The load-bearing mechanism is one dual-encoder that receives three training signals. It is first pre-trained as Legal-SCP with a self-supervised masked auto-encoding objective over adjacent spans of legal text, which adapts the encoder to legal vocabulary and discourse. It is then fine-tuned with Circle Loss on question-answer pairs, where the hard negatives are top-ranked answers retrieved by BM25 that are not the labeled positive. Finally, the re-ranking stage computes, for each candidate question, an affinity vector of dot products against eight anchor questions, passes the projected affinity vectors through a Transformer, and optimizes a contrastive loss together with an MSE loss that keeps the refined representations close to the original affinity features. This affinity aggregation is what lets a short user query borrow context from its most similar questions.","core_discovery":"The paper's central claim is that the three stages are complementary and that their combination produces a legal QA retriever that outperforms existing dense-retrieval methods. Stage one pre-trains an encoder-decoder, Legal-SCP, on legal question and answer spans with masked auto-encoding; stage two keeps only the encoder and fine-tunes it on lawyer-annotated QA pairs using Circle Loss with BM25-mined hard negatives; stage three represents each candidate question by an affinity vector of its dot products with eight anchor questions, refines these vectors with a Transformer under a contrastive loss, and preserves the original affinity information with an MSE term. On the LawQA test set the complete model reaches 79.9 P@1 and 87.3 MRR@16, exceeding SimCSE by 5.7 and 4.2 points respectively; the ablations show task-specific fine-tuning is the largest single contributor.","pith_inferences":["Editorial inference: because the test set contains only 900 pairs, the 5.7-point P@1 gap corresponds to about 51 questions, so the reported advantage should be treated as provisional until significance or a second test set is reported.","Editorial inference: the re-ranking stage consumes only candidate questions and their dot-product affinities, so it could be attached to any dense retriever; the paper does not test that combination.","Editorial inference: the gains are measured on a legal advice forum, and may not transfer to formal legal search over statutes or case law; a test on court-document queries would mark that boundary."],"forward_implications":["A deployed legal QA system using PFR-LQA would rank the correct answer first for 79.9% of LawQA-style queries, up from 74.2% with SimCSE.","Task-specific fine-tuning carries the largest share of the gain: removing it drops P@1 from 79.9 to 73.7, lower than removing either of the other two stages.","Domain-specific pre-training contributes more than contextual re-ranking, since removing it drops P@1 to 75.8 while removing re-ranking drops it to 78.5.","All three stages are complementary: the full model beats every two-stage ablation on both P@1 and MRR@16."],"supporting_citations":[{"why":"It supplies the contextual mask auto-encoder pre-training objective that Legal-SCP adapts to legal span pairs.","marker":"[15]"},{"why":"It supplies the Circle Loss used to fine-tune the dual-encoder on positive and hard-negative question-answer pairs.","marker":"[16]"},{"why":"It supplies the InsuranceQA-style evaluation scheme that motivates the LawQA train-validation-test construction with BM25 negatives.","marker":"[17]"},{"why":"It provides LawFormer, a legal-domain pre-trained baseline that the method must outperform.","marker":"[18]"},{"why":"It provides DPR, the dual-encoder dense retrieval baseline that motivates the encoder architecture.","marker":"[19]"},{"why":"It provides ColBERT, a late-interaction retrieval baseline that is compared against on LawQA.","marker":"[20]"},{"why":"It provides SimCSE, the strongest baseline that PFR-LQA claims to beat by 5.7 P@1 and 4.2 MRR@16.","marker":"[21]"},{"why":"It provides BERT, the base model whose fine-tuned performance anchors the lower end of the baseline range.","marker":"[11]"},{"why":"It provides RoBERTa, the model that initializes the Legal-SCP encoder before domain pre-training.","marker":"[12]"}],"fun_headline_variants":["Legal QA: three stages beat SimCSE by 5.7","Pretrain, finetune, rerank: legal QA recipe beats SimCSE","Fine-tuning key to beating SimCSE in legal QA","Legal QA triple play: pretrain, finetune, rerank hits 79.9","Legal QA: PFR-LQA beats SimCSE by 5.7 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 900 LawQA test pairs are a fair, representative sample of legal queries; the paper reports no significance tests, and the 5.7-point P@1 gap corresponds to roughly 51 questions, so the advantage could in principle be sampling noise.","fun_headline_variants_meta":{"raw":{"variants":["Legal QA: three stages beat SimCSE by 5.7","Pretrain, finetune, rerank: legal QA recipe beats SimCSE","Fine-tuning key to beating SimCSE in legal QA","Legal QA triple play: pretrain, finetune, rerank hits 79.9","Legal QA: PFR-LQA beats SimCSE by 5.7 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002028,"raw_usage":{"total_tokens":7934,"prompt_tokens":1004,"completion_tokens":6930,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":6826}},"tokens_in":620,"tokens_out":6930,"duration_ms":44359,"temperature":1.0,"reasoning_tokens":6826,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:17:05.943432+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Bootstrap-resample the 900 LawQA test pairs from the full positive-pair pool and compute a 95% confidence interval for the P@1 difference between PFR-LQA and SimCSE; if the interval contains zero, the claimed 5.7-point superiority is not established.","supporting_citations":[{"cited_title":"Contextual mask auto-encoder for dense passage retrieval,","cited_arxiv_id":null,"evidence_quote":"It supplies the contextual mask auto-encoder pre-training objective that Legal-SCP adapts to legal span pairs."},{"cited_title":"Circle loss: A unified perspective of pair similarity optimization,","cited_arxiv_id":null,"evidence_quote":"It supplies the Circle Loss used to fine-tune the dual-encoder on positive and hard-negative question-answer pairs."},{"cited_title":"Applying deep learning to answer selection: A study and an open task,","cited_arxiv_id":null,"evidence_quote":"It supplies the InsuranceQA-style evaluation scheme that motivates the LawQA train-validation-test construction with BM25 negatives."},{"cited_title":"Lawformer: A pre-trained language model for chinese legal long documents,","cited_arxiv_id":null,"evidence_quote":"It provides LawFormer, a legal-domain pre-trained baseline that the method must outperform."},{"cited_title":"Dense passage retrieval for open-domain question an- swering,","cited_arxiv_id":null,"evidence_quote":"It provides DPR, the dual-encoder dense retrieval baseline that motivates the encoder architecture."},{"cited_title":"SimCSE: Simple contrastive learning of sentence embeddings,","cited_arxiv_id":null,"evidence_quote":"It provides SimCSE, the strongest baseline that PFR-LQA claims to beat by 5.7 P@1 and 4.2 MRR@16."},{"cited_title":"Roberta: A robustly optimized bert pretraining approach,","cited_arxiv_id":null,"evidence_quote":"It provides RoBERTa, the model that initializes the Legal-SCP encoder before domain pre-training."}],"review_version":1}