REVIEW 4 major objections 5 minor 24 references
Pre-training, Fine-tuning and Re-ranking: A Three-Stage Framework for Legal Question Answering
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Three-stage legal QA framework—domain pre-training, fine-tuning, and contextual re-ranking—beats the strongest baseline by 5.7 points P@1.
desk verdict A sensible three-stage legal QA pipeline with a potentially valuable dataset, but the evaluation rests on a 900-pair test set whose construction and split unit are unspecified, so the headline gains aren't yet trustworthy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is one dual-encoder that receives three training signals. It is first pre-trained as Legal-SCP with a self-supervised masked auto-encoding objective over adjacent spans of legal text, which adapts the encoder to legal vocabulary and discourse. It is then fine-tuned with Circle Loss on question-answer pairs, where the hard negatives are top-ranked answers retrieved by BM25 that are not the labeled positive. Finally, the re-ranking stage computes, for each candidate question, an affinity vector of dot products against eight anchor questions, passes the projected affinity vectors through a Transformer, and optimizes a contrastive loss together with an MSE loss that keeps the refined representations close to the original affinity features. This affinity aggregation is what lets a short user query borrow context from its most similar questions.
What would settle it
Bootstrap-resample the 900 LawQA test pairs from the full positive-pair pool and compute a 95% confidence interval for the P@1 difference between PFR-LQA and SimCSE; if the interval contains zero, the claimed 5.7-point superiority is not established.
Extended reading notes
Core claim
The paper's central claim is that the three stages are complementary and that their combination produces a legal QA retriever that outperforms existing dense-retrieval methods. Stage one pre-trains an encoder-decoder, Legal-SCP, on legal question and answer spans with masked auto-encoding; stage two keeps only the encoder and fine-tunes it on lawyer-annotated QA pairs using Circle Loss with BM25-mined hard negatives; stage three represents each candidate question by an affinity vector of its dot products with eight anchor questions, refines these vectors with a Transformer under a contrastive loss, and preserves the original affinity information with an MSE term. On the LawQA test set the complete model reaches 79.9 P@1 and 87.3 MRR@16, exceeding SimCSE by 5.7 and 4.2 points respectively; the ablations show task-specific fine-tuning is the largest single contributor.
Load-bearing premise
The load-bearing premise is that the 900 LawQA test pairs are a fair, representative sample of legal queries; the paper reports no significance tests, and the 5.7-point P@1 gap corresponds to roughly 51 questions, so the advantage could in principle be sampling noise.
Editorial extensions
If this is right
- A deployed legal QA system using PFR-LQA would rank the correct answer first for 79.9% of LawQA-style queries, up from 74.2% with SimCSE.
- Task-specific fine-tuning carries the largest share of the gain: removing it drops P@1 from 79.9 to 73.7, lower than removing either of the other two stages.
- Domain-specific pre-training contributes more than contextual re-ranking, since removing it drops P@1 to 75.8 while removing re-ranking drops it to 78.5.
- All three stages are complementary: the full model beats every two-stage ablation on both P@1 and MRR@16.
Reading between the lines
- Editorial inference: because the test set contains only 900 pairs, the 5.7-point P@1 gap corresponds to about 51 questions, so the reported advantage should be treated as provisional until significance or a second test set is reported.
- Editorial inference: the re-ranking stage consumes only candidate questions and their dot-product affinities, so it could be attached to any dense retriever; the paper does not test that combination.
- Editorial inference: the gains are measured on a legal advice forum, and may not transfer to formal legal search over statutes or case law; a test on court-document queries would mark that boundary.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PFR-LQA, a three-stage framework for Chinese legal question answering. Stage one pre-trains a dual-encoder model (Legal-SCP) on legal questions and answers with self-supervised and context-supervised masked auto-encoding objectives; stage two fine-tunes the encoder on question-answer pairs with circle loss; stage three re-ranks candidates by computing affinity features against anchors and refining them with a Transformer, using a contrastive loss plus an MSE reconstruction loss. The authors construct a large Chinese legal QA dataset called LawQA with 549,668 positive QA pairs and report experiments comparing PFR-LQA with BM25, BERT, RoBERTa, LawFormer, DPR, ColBERT, and SimCSE. On a 900-pair test set, the proposed method achieves P@1 of 79.9 and MRR@16 of 87.3, outperforming the best baseline SimCSE by 5.7 and 4.2 points, respectively. Three ablation variants (w/o DSP, w/o TSF, w/o CR) are included in Table II.
Significance. If the reported results are reliable, PFR-LQA would demonstrate that combining domain-specific pre-training, task-specific dense retrieval fine-tuning, and contextual re-ranking yields a meaningful advance for community-style legal question answering, a practically important task. The construction of a large legal QA corpus with lawyer-written answers is also a potentially useful resource for the community, provided the dataset is released with a clear and leakage-safe evaluation protocol. The paper's high-level idea is reasonable and builds on established components (Legal-SCP, circle loss, and similarity aggregation), but the central empirical claim is currently not supported at the level of rigor expected for a state-of-the-art claim: the evaluation lacks statistical validation, the test set construction is underspecified, and the re-ranking stage contains an underspecified threshold-based labeling procedure.
major comments (4)
- [III-A and III-D] The construction and sampling of the 900-pair test set are not described, and the split unit is never specified. Table I reports category sizes in terms of 'cases' (322,475 total), while the text reports 549,668 positive QA pairs, which implies that a single case or user query can have multiple gold answers. If the 900 test pairs were sampled at the QA-pair level rather than at the query level, the same query can appear in both training and test folds, and the reported P@1 gap of 5.7 points corresponds to only about 51 test pairs. The authors should specify the split unit, perform a query-level split, report train/test overlap statistics, and provide confidence intervals or significance tests for the pairwise differences in Table II. Without this, the claimed advantage over SimCSE is indistinguishable from a test-construction artifact.
- [II-C and III-C] The positive/negative labeling used to train the contextual re-ranking stage is self-referential and is not fully specified. The paper states that 'question pairs with small similarities (less than a threshold)' are treated as negative samples and the remaining retrieved candidates as positive samples, but the threshold value is not given, and no justification is provided for treating all remaining candidates as positive. In addition, the procedure for selecting the L anchor questions from the K candidates is not described. These details are load-bearing because the re-ranking stage is one of the three claimed contributions, and the choice of threshold can determine whether the stage learns genuinely new information or merely re-ranks according to the fine-tuned model's own similarities.
- [III-D and III-E] No measure of variability is reported anywhere in the experiments. There are no standard deviations over multiple runs, no random seeds, and no significance tests for the differences in Table II or for the ablation rows. Given that the test set has only 900 pairs, a 5.7-point P@1 gap is about 51 questions, and the ablation gaps (e.g., 4.2 points between PFR-LQA and w/o DSP) are of similar magnitude. The authors should report confidence intervals, bootstrap estimates, or significance tests before claiming that the three-stage framework yields substantial improvements.
- [III-E] The ablation study is described in a single sentence and does not specify how each component is removed. For instance, 'w/o DSP' might mean starting from RoBERTa without legal-domain pre-training, and 'w/o CR' might mean returning the BM25 top-K candidates without re-ranking, but the text does not say. It is also unclear whether hyperparameters are re-tuned for each ablation. Since the paper's contribution is explicitly the collaborative effect of the three stages, the ablation must state what exactly is removed and how the remaining pipeline is retrained; otherwise the relative contribution of each stage cannot be assessed.
minor comments (5)
- [Section headings] The section numbering is inconsistent: 'III. EXPERIMENTS' is immediately followed by 'IV. EXPERIMENTAL SETUP', and the experimental content actually appears under the latter heading. The headings should be renumbered.
- [III-A] Please clarify the relation between the 549,668 positive QA pairs and the 322,475 total 'cases' in Table I. The table caption and the surrounding text use different units, and the number of QA pairs per case is not stated.
- [III-A] The phrase 'randomly select several negative candidates via the BM25 algorithm' is vague. The number of negative candidates sampled per positive pair and the size of the BM25 candidate pool should be stated, especially because this sampling determines the training and validation set sizes.
- [II-C, Eq. (4)] The index range '1 ≤ i ≤ L ≤ K' appears to contain a typo, since the equation uses j for the candidate question and i does not appear elsewhere; it should likely be '1 ≤ j ≤ L ≤ K'.
- [Throughout] There are several typos and stylistic issues, including 'fune-tuning' in Section IV-C, 'peform' in Section III-E, and 'we use the pre-trained Legal-SCP model' in Section II-C, which should be 'the fine-tuned Legal-SCP model' for clarity.
Circularity Check
Minor self-referential training signal in the re-ranking stage; the central test-set evaluation remains externally grounded.
-
other
[Section II-C, Question Re-ranking with Similarity Aggregation (around Eq. 6)]
"Note that we use the fine-tuned dual-encoder retrieval model to decide the positive and negative samples by calculating question similarities. That is, the question pairs with small similarities (less than a threshold) are treated as negative samples, and the remaining retrieved candidate questions are regarded as positive samples."
The re-ranking contrastive loss in Eq. (6) is optimized against positive/negative sets that are not defined by external gold relevance judgments but by the fine-tuned model's own cosine-similarity threshold. Thus the 'contextual re-ranking' module is trained to sharpen a separation already made by the model it refines; the final ranking is a learned function of that model's prior similarity scores. This is a self-referential training signal rather than an independent derivation: the re-ranker's notion of 'positive' is whatever the fine-tuned encoder already regards as similar. The test evaluation still uses external labels, so the central performance claim is not fully circular, but the improvement attributed to re-ranking is partly a self-distillation/bootstrap effect.
full rationale
The paper's central claim is an empirical result measured on a test set with gold labels, so it does not reduce to its own inputs by construction. The clearest self-referential element is in Section II-C, where the re-ranking model's positive/negative examples are generated by the fine-tuned encoder's own similarity scores; this is a training-signal issue rather than a derivation-equivalence. No load-bearing self-citation was found: the cited prior work (e.g., Contextual Mask Auto-Encoder, Circle Loss, InsuranceQA) is external and not used as a uniqueness theorem or as the sole justification for the method. I also note non-circular correctness risks that should be weighed separately: Section III-A does not specify whether the 900 test QA pairs are split at the user-query level or the QA-pair level, Section III-D reports no significance tests or confidence intervals for the 5.7-point P@1 gap, and Section III-C says the pre-training stage adopts 'the QA pairs in the entire database,' which may include the test pairs. These concerns affect evaluation validity and generalizability, but they are not examples of the derivation being equivalent to its input by construction, so they do not raise the circularity score beyond 2.
Assumptions & free parameters
free parameters (3)
- Positive/negative sampling threshold in contextual re-ranking
- Number of anchors L and candidates K =
L=8, K=16
- Loss weighting lambda =
lambda=0.2
assumptions (3)
- domain assumption The LawQA dataset is representative and the 900-pair test set is an unbiased sample of the legal QA distribution.
- domain assumption BM25 top-16 retrieval has sufficient recall to include the correct answer/question for the re-ranking stages.
- domain assumption Similarity between queries reflects answer relevance.
Cite this review
Pith. "Pith review of Pre-training, Fine-tuning and Re-ranking: A Three-Stage Framework for Legal Question Answering." pith.science (2026). https://pith.science/paper/S43EPMVJ
@misc{pith2026241219482,
author = {Pith},
title = {Pith review of: Pre-training, Fine-tuning and Re-ranking: A Three-Stage Framework for Legal Question Answering},
year = {2026},
howpublished = {\url{https://pith.science/paper/S43EPMVJ}},
note = {Machine review of arXiv:2412.19482}
}
read the original abstract
Legal question answering (QA) has attracted increasing attention from people seeking legal advice, which aims to retrieve the most applicable answers from a large-scale database of question-answer pairs. Previous methods mainly use a dual-encoder architecture to learn dense representations of both questions and answers. However, these methods could suffer from lacking domain knowledge and sufficient labeled training data. In this paper, we propose a three-stage (\underline{p}re-training, \underline{f}ine-tuning and \underline{r}e-ranking) framework for \underline{l}egal \underline{QA} (called PFR-LQA), which promotes the fine-grained text representation learning and boosts the performance of dense retrieval with the dual-encoder architecture. Concretely, we first conduct domain-specific pre-training on legal questions and answers through a self-supervised training objective, allowing the pre-trained model to be adapted to the legal domain. Then, we perform task-specific fine-tuning of the dual-encoder on legal question-answer pairs by using the supervised learning objective, leading to a high-quality dual-encoder for the specific downstream QA task. Finally, we employ a contextual re-ranking objective to further refine the output representations of questions produced by the document encoder, which uses contextual similarity to increase the discrepancy between the anchor and hard negative samples for better question re-ranking. We conduct extensive experiments on a manually annotated legal QA dataset. Experimental results show that our PFR-LQA method achieves better performance than the strong competitors for legal question answering.
Figures
Reference graph
Works this paper leans on
-
[4]
Re-ranking answer selection with similarity aggregation,
D. Chen, S. Peng, K. Li, Y . Xu, J. Zhang, and X. Xie, “Re-ranking answer selection with similarity aggregation,” in Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020, pp. 1677–1680
work page 2020
-
[1]
Re- trieving and reading: A comprehensive survey on open-domain question answering,
F. Zhu, W. Lei, C. Wang, J. Zheng, S. Poria, and T.-S. Chua, “Re- trieving and reading: A comprehensive survey on open-domain question answering,” arXiv preprint arXiv:2101.00774 , 2021
arXiv 2021
-
[2]
Experiments with convolutional neural network models for answer selection,
J. Rao, H. He, and J. Lin, “Experiments with convolutional neural network models for answer selection,” in Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2017, pp. 1217–1220
work page 2017
-
[3]
Knowledge-aware attentive neural network for ranking question answer pairs,
Y . Shen, Y . Deng, M. Yang, Y . Li, N. Du, W. Fan, and K. Lei, “Knowledge-aware attentive neural network for ranking question answer pairs,” in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , 2018, pp. 901–904
work page 2018
-
[5]
Reranking for efficient transformer-based answer selection,
Y . Matsubara, T. Vu, and A. Moschitti, “Reranking for efficient transformer-based answer selection,” in Proceedings of the 43rd in- ternational ACM SIGIR conference on research and development in information retrieval, 2020, pp. 1577–1580
work page 2020
-
[6]
F. Kunneman, T. C. Ferreira, E. Krahmer, and A. Van Den Bosch, “Question similarity in community question answering: A systematic exploration of preprocessing methods and models,” in Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019) , 2019, pp. 593–601
work page 2019
-
[7]
Jaist: Combining multiple features for answer selection in community question answering,
Q. H. Tran, D.-V . Tran, T. Vu, M. Le Nguyen, and S. B. Pham, “Jaist: Combining multiple features for answer selection in community question answering,” in Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015) , 2015, pp. 215–219
work page 2015
-
[8]
Automatic feature engineering for answer selection and extraction,
A. Severyn and A. Moschitti, “Automatic feature engineering for answer selection and extraction,” in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , 2013, pp. 458– 467
work page 2013
Show all 24 references
-
[9]
Few- shot question answering by pretraining span selection,
O. Ram, Y . Kirstain, J. Berant, A. Globerson, and O. Levy, “Few- shot question answering by pretraining span selection,” arXiv preprint arXiv:2101.00438, 2021
2021 arXiv
-
[10]
Deep learning-based question answering: a survey,
H. Abdel-Nabi, A. Awajan, and M. Z. Ali, “Deep learning-based question answering: a survey,” Knowledge and Information Systems, pp. 1–87, 2022
2022
-
[11]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, 2019, pp. 4171–4186
2019
-
[12]
Roberta: A robustly optimized bert pretraining approach,
Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” ICLR, 2020
2020
-
[13]
Span selection pre-training for question answer- ing,
M. Glass, A. Gliozzo, R. Chakravarti, A. Ferritto, L. Pan, G. Bhargav, D. Garg, and A. Sil, “Span selection pre-training for question answer- ing,” arXiv preprint arXiv:1909.04120 , 2019
1909 arXiv
-
[14]
Question answering infused pre- training of general-purpose contextualized representations,
R. Jia, M. Lewis, and L. Zettlemoyer, “Question answering infused pre- training of general-purpose contextualized representations,” in Findings of the Association for Computational Linguistics , 2022, pp. 711–728
2022
-
[15]
Contextual mask auto-encoder for dense passage retrieval,
X. Wu, G. Ma, M. Lin, Z. Lin, Z. Wang, and S. Hu, “Contextual mask auto-encoder for dense passage retrieval,” AAAI, 2023
2023
-
[16]
Circle loss: A unified perspective of pair similarity optimization,
Y . Sun, C. Cheng, Y . Zhang, C. Zhang, L. Zheng, Z. Wang, and Y . Wei, “Circle loss: A unified perspective of pair similarity optimization,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 6398–6407
2020
-
[17]
Applying deep learning to answer selection: A study and an open task,
M. Feng, B. Xiang, M. R. Glass, L. Wang, and B. Zhou, “Applying deep learning to answer selection: A study and an open task,” in 2015 IEEE workshop on automatic speech recognition and understanding (ASRU) . IEEE, 2015, pp. 813–820
2015
-
[18]
Lawformer: A pre-trained language model for chinese legal long documents,
C. Xiao, X. Hu, Z. Liu, C. Tu, and M. Sun, “Lawformer: A pre-trained language model for chinese legal long documents,” AI Open, vol. 2, pp. 79–84, 2021
2021
-
[19]
Dense passage retrieval for open-domain question an- swering,
V . Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question an- swering,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , 2020, pp. 6769–6781
2020
-
[20]
Colbert: Efficient and effective passage search via contextualized late interaction over bert,
O. Khattab and M. Zaharia, “Colbert: Efficient and effective passage search via contextualized late interaction over bert,” in Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval , 2020, pp. 39–48
2020
-
[21]
SimCSE: Simple contrastive learning of sentence embeddings,
T. Gao, X. Yao, and D. Chen, “SimCSE: Simple contrastive learning of sentence embeddings,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021, pp. 6894– 6910
2021
-
[22]
The stem cell hypothesis: Dilemma behind multi- task learning with transformer encoders,
H. He and J. D. Choi, “The stem cell hypothesis: Dilemma behind multi- task learning with transformer encoders,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021, pp. 5555–5577
2021
-
[23]
Decoupled weight decay regularization,
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017
2017 arXiv
-
[24]
Stochastic gradient descent tricks,
L. Bottou, “Stochastic gradient descent tricks,” Neural Networks: Tricks of the Trade: Second Edition , pp. 421–436, 2012
2012
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.