Pith. sign in

REVIEW 3 major objections 6 minor 18 references

DRAssist: Dispute Resolution Assistance using Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Asking an LLM to grade each argument strong or weak before naming the stronger party and ruling on demands markedly improves dispute assistance, lifting macro-F1 from 0.53 to 0.78 on automobile-insurance disputes.

desk verdict A cleanly written prototype for LLM-assisted dispute resolution with two new datasets, but the headline CoT result is undercut by gold labels generated by the same model family being tested. read the letter →

arxiv 2509.01962 v1 pith:MOJNAZWV submitted 2025-09-02 cs.CL

Sachin Pawar , Manoj Apte , Girish K. Palshikar , Basit Ali , Nitin Ramrakhiyani This is my paper · ORCID
classification cs.CL
keywords disputeresolutionlargelanguagemodelschain-of-thoughtpromptingzero-shotreasoningargumentstrengthevaluationstructuredsummarizationinsurancedisputesdomainname
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that large language models can genuinely help a human judge or arbitrator prepare a dispute resolution, provided the model works through the dispute in the right order. DRAssist first converts an unstructured dispute text into a fixed structure — agreed facts, points of disagreement, each side's arguments and demands — and then prompts an LLM with three strategies; the finding is that the chain-of-thought strategy S3 (score each argument strong or weak first, then name the overall stronger party and rule each demand) beats the two simpler strategies on both test domains. On automobile-insurance disputes, the S3 ensemble reaches macro-F1 — the balanced average of precision and recall across the two outcome classes — of 0.78, against 0.53 for the best baseline; on domain-name disputes the best S3 model reaches 0.62 versus 0.46, with argument-wise evaluation at 0.73. The paper frames this deliberately as assistance, not automation: the output is an explained first-cut recommendation for a human decision-maker, aimed at reducing the effort, time, and subjectivity in high-volume pre-legal forums. If the claim holds, useful adjudication support is available without training on any corpus of past disputes.

What carries the argument

The load-bearing object is the S3 prompt, a zero-shot chain-of-thought template: it instructs the model to (1) rewrite and label every argument of both parties as STRONG or WEAK, (2) identify the overall stronger party from those evaluations, and (3) label each demand ACCEPTED or REJECTED, with a consistency rule barring opposing demands from sharing a label. The two weaker strategies — S1 (stronger party only) and S2 (demand rulings without argument scoring) — isolate the contribution of the intermediate argument-evaluation step. The other load-bearing piece is the structured summary: each dispute is converted, element by element (agreed facts, disagreement aspects, arguments, demands, prio

What would settle it

Have adjudicators — retired consumer-forum members for the insurance disputes, trademark and arbitration lawyers for the domain-name disputes — label the winning party, each demand, and each argument as strong or weak on a random sample of 50–100 disputes per dataset, then recompute accuracy and macro-F1 for S1, S2, and S3 against those human labels. If the S3 advantage over S2 and S1 collapses, or if S3 matches the LLM-generated gold labels while disagreeing with the human ones, the central claim is refuted. Have the same humans rate the arguments separately to locate where disagreement origi

Watch

Extended reading notes

Core claim

The paper's central claim: the chain-of-thought prompting strategy, S3, gives the best dispute-resolution assistance of the three strategies tested. To use S3, the LLM first labels each argument of each party STRONG or WEAK, then names the overall stronger party and rules each demand ACCEPTED or REJECTED, barring conflicting demands from the same label. For stronger-party prediction, the ordering S3 > S2 > S1 holds on both datasets: macro-F1 0.78 vs 0.53 baseline on automobile-insurance disputes, and 0.62 vs 0.46 on domain-name disputes; under S3, argument evaluation reaches 0.73 and demand rulings 0.64. A further result: despite predicting better, S3's justifications score lower on ROUGE an

Load-bearing premise

The labels that the evaluation scores the models against are produced by the same family of models being tested: GPT-4o-mini is prompted with the original dispute text — which already contains the final decision — to mark each demand and argument, and the winning-party gold standard is a majority vote across LLM-generated summaries that include GPT-4o-mini (Section 5.1); if those labels diverge from what human adjudicators would decide, the reported accuracies measure self-co

Editorial extensions

If this is right

  • Useful first-cut adjudication support becomes available with zero supervision: no corpus of past cases is needed for training, which matters for high-volume pre-legal forums where labeled data is scarce.
  • The consistency constraint in the S3 prompt (conflicting demands cannot both be ACCEPTED) shows that structurally sound resolutions can be coaxed from an LLM by prompt design rather than by post-hoc repair.
  • Argument-wise STRONG/WEAK ratings are a usable byproduct: parties could receive a per-argument strength breakdown with justifications, independent of the ultimate verdict.
  • Because S3 predicts better but justifies worse than S1/S2, a practical system would run S3 for the prediction and a separate direct prompt for the narrative rationale, trading a little accuracy for readable explanations.
  • The paper's error analysis implies the summarization stage sets a ceiling on resolution accuracy — errors in the structured summary propagate into wrong verdicts — so improving the upstream summary is a direct lever on downstream performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the gold labels for demands and arguments are produced by prompting GPT-4o-mini on text that already contains the final decision, and the winning-party gold standard is a majority of LLM summaries that include the same model family, the reported F1 scores are best read as measures of model self-consistency; they become statements about agreement with human adjudication only after re-scoring
  • If the S3 advantage survives human labels, the likely reason is that structuring the question — not any domain tuning — drives the gain, so the same template should transfer to other adjudicative text domains such as tax, banking, or health-insurance complaints; testing this on one additional corpus would be a cheap validation.
  • The strong complainant bias observed in the domain-name data (recall for the respondent as low as 0.31 even in the best setting) suggests the models lean on a pretraining prior — complainants usually win in such disputes — rather than case content; a party-label swap (keep arguments, exchange 'complainant' and 'respondent' names) would separate label-prior from content-based reasoning.
  • The paper leaves open whether the structured summary is what carries the accuracy: running the same S3 prompt directly on the raw dispute text would isolate the contribution of the summarization scaffold, and the error analysis predicts accuracy would drop if summaries are imperfect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents DRAssist, a prototype dispute-resolution assistance system that uses large language models in a zero-shot manner. It contributes two datasets (104 auto-insurance disputes from NCDRC, 351 domain-name disputes from WIPO), an LLM-based structured summary for each dispute, and three LLM prompting strategies S1 (direct stronger-party prediction), S2 (stronger party plus demand-wise decisions), and S3 (chain-of-thought: argument-strength evaluation followed by stronger-party and demand-wise predictions). The authors evaluate Mistral-7B, Llama-3-8B, and GPT-4o-mini, plus a majority-vote ensemble, reporting accuracy and macro-F1 for stronger-party prediction, demand-wise decisions, and argument-wise evaluation. The main claim is that S3 outperforms S2, which outperforms S1, and that S3 substantially beats majority/random baselines, e.g., macro-F1 0.78 vs. 0.53 for stronger-party prediction on the auto-insurance dataset. The paper also reports summarization quality via ROUGE/BERTScore against noisy rhetorical-role references and analyzes a few error cases.

Significance. If the reported results were grounded in human-validated ground truth, the paper would be a useful demonstration that zero-shot LLMs can provide multi-level assistance in dispute resolution, with the notable strengths of introducing two new domain datasets, releasing structured summaries and ground-truth annotations, and comparing several LLMs and prompting strategies. The error analysis in Section 5.6 is valuable and honest about failure modes. However, the central evaluation is currently not trustworthy for real adjudication: the demand-wise and argument-wise gold labels are generated by GPT-4o-mini itself, and the winning-party gold for the auto-insurance dataset is derived from LLM summaries. The reported F1 scores therefore largely measure self-consistency between an LLM annotator and LLM predictors from the same family. The paper explicitly defers a human user study to future work. Because the quantitative claims—especially the headline S3 advantage over S1/S2—rest on this circular evaluation, the contribution in its present form is a promising pipeline and dataset, not yet a validated demonstration of dispute-resolution assistance.

major comments (3)
  1. The demand-wise and argument-wise gold labels are created by prompting GPT-4o-mini with the original dispute text that contains the final decision, asking it to infer per-demand acceptance and per-argument judge favorability. GPT-4o-mini is also one of the three evaluated LLMs. Consequently, the reported demand/argument accuracies and macro-F1 values (e.g., D_AI argument-wise S3 Ensemble macro-F1 0.60, Table 6) measure agreement between GPT-4o-mini as annotator and GPT-4o-mini as predictor, not agreement with any human judge. This circularity can also inflate the relative advantage of S3, because the S3 prompt asks the model to label each argument STRONG/WEAK using criteria similar to those in the gold-generation prompt. To support the paper's claims, the authors need human-validated gold labels on a sample (or an independent set of human decisions) and should report agreement between LL
  2. For D_AI, the winning-party gold standard is obtained by majority vote among three LLM-generated structured summaries (Mistral, Llama, GPT-4o-mini), with no human verification. The final decision text is included in the original dispute, but the summaries themselves may hallucinate or omit the winner. Since the headline result for stronger-party prediction (S3 Ensemble macro-F1 0.78 on D_AI) is computed against this LLM-derived gold, an incorrect winner in even a modest fraction of the 104 disputes would materially change the reported accuracy. The D_DN winning-party gold is extracted by regex from the WIPO decisions and is more reliable, but it is not validated either. The authors should verify the winning-party labels against the original decisions by human inspection or by extracting from the explicitly structured 'Decision' sections.
  3. The paper claims that 'S3 is better than S2 and S2 is better than S1 for both the datasets,' but no significance testing or confidence intervals are reported. Several differences in Table 6 are within sampling noise: for D_AI demand-wise decisions, S2 Ensemble macro-F1 is 0.63 and S3 Ensemble is 0.62; for D_DN stronger-party prediction, S3 Ensemble macro-F1 is 0.52, which is worse than S3 Llama (0.62) and comparable to S2 GPT (0.53). Point estimates on 104 or 351 disputes cannot support a strict ordering, especially for differences of 0.01–0.03. The authors should report paired significance tests (e.g., McNemar's test or bootstrap confidence intervals) and effect sizes for the S3-vs-S2 and S2-vs-S1 comparisons.
minor comments (6)
  1. [Abstract] The phrase 'aspects or disagreement' should be 'aspects of disagreement' or 'disagreement aspects'.
  2. [Section 4.1 and Table 3] The summarization evaluation uses rhetorical-role labels from 'opennyai' as references; the paper acknowledges these labels are imperfect. Given the low ROUGE scores (e.g., R1-F1 0.30–0.42), the summarization quality claim should be framed as approximate and perhaps supplemented with a small human evaluation of sampled structured summaries.
  3. [Section 5.2] Justification quality is computed only for disputes where the stronger-party prediction is correct. This selection bias should be stated more prominently, as it may overstate justification quality on easier disputes and makes cross-strategy comparisons of justification metrics difficult to interpret.
  4. [Section 5.4] The ensemble mapping for demand/argument labels relies on one-to-one matching to the ground truth; errors in this matching propagate to the ensemble vote. The paper should state how ties in majority voting are broken and what happens when one LLM omits a demand/argument entirely.
  5. [Section 5.1] The cosine-similarity matching uses a linear sum assignment algorithm, but no similarity threshold is reported. Very low-similarity matches could create spurious alignments between the LLM output and ground truth; an analysis or threshold would strengthen the evaluation.
  6. [Table 6] The random baseline macro-F1 values are not 0.5 in several rows (e.g., 0.53 for D_AI stronger-party). Since random labels are class-dependent, state the random-label generation procedure and report variance over multiple runs.

Circularity Check

2 steps flagged · score 6.0 of 10

Evaluation circularity: demand/argument gold labels are generated by GPT-4o-mini, and the D_AI winning-party gold is a majority vote of the same three LLMs later ensembled.

  1. self definitional [Section 5.1 (Ground truth creation), used in Table 6]
    "In order to create the ground truth for the demand-wise, we prompt the GPT-4o-mini model with the original dispute text (which contains the final decision on the dispute) along with the list of demands of a party (identified as part of the structured summary) to obtain the true decision for each demand, i.e., whether each demand was accepted or rejected in the final decision. Similarly, we obtain ground truth labels for each argument of each party depending on whether that particular argument was considered favourably by the judge while arriving at the final decision."

    GPT-4o-mini is one of the three evaluated LLMs (Section 5). The gold labels for demand-wise decisions and argument-wise evaluation are produced by prompting GPT-4o-mini with the final decision. Therefore, when Table 6 reports GPT-4o-mini's demand/argument F1, it measures agreement between GPT-4o-mini-as-annotator and GPT-4o-mini-as-predictor. This is a self-consistency check, not an independent human gold standard. The claimed S3 advantage in argument reasoning is partly evaluated against labels generated by the same model family, so part of the reported improvement may reflect prompt alignment rather than adjudication quality.

  2. self definitional [Section 4.1 (Structured Summarization) and Section 5.5 (Evaluation Results)]
    "For deciding the winning party in the super summary, we consider the majority vote between the winning parties identified by the 3 individual summaries. ... For D_AI there is a big improvement over the baselines, as S3 (Ensemble) achieves macro-F1 of 0.78 as compared to the baseline performance of 0.53."

    The '3 individual summaries' are generated by the same three LLMs later evaluated: Mistral, Llama-3-8B, and GPT-4o-mini (Sections 4.1 and 5). Thus the D_AI winning-party gold standard is a majority vote of the three test models' own structured summaries. The headline S3 Ensemble result is also a majority vote across the same three models' predictions. The 0.78 macro-F1 is therefore partly the ensemble agreeing with a gold label produced by a majority of the same models, with no independent human decision in the loop. This contaminates the paper's central quantitative claim.

full rationale

The paper's actual derivation pipeline—structured summary, S1/S2/S3 prompting, prediction—is not circular in the sense that the output equals the input. The problem is the gold-standard construction. For demand-wise and argument-wise evaluation, the gold labels are created by prompting GPT-4o-mini with the original dispute text containing the final decision (Section 5.1). Since GPT-4o-mini is one of the evaluated models, its reported F1 scores are partly self-agreement between the same model acting as annotator and as predictor. For the D_AI stronger-party task, the gold winning party is the majority vote of structured summaries generated by the same three LLMs that are later ensembled (Section 4.1), so the headline S3 Ensemble macro-F1 of 0.78 is partly agreement with a self-generated label. The paper's own conclusion defers a human user study to future work, confirming that no independent adjudication benchmark is used. For the non-GPT models (Mistral, Llama) the demand/argument labels are not self-generated, and the D_DN winning-party gold is regex-derived, so the central S3-over-baseline ordering retains some independent content. Nevertheless, the absolute numbers and the GPT-4o-mini comparisons are contaminated by construction, warranting partial circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central evaluation rests on LLM-generated labels and summaries as ground truth, plus the assumptions that binary labels and zero-shot world knowledge are adequate. Since GPT-4o-mini both produces the gold labels for demands/arguments and is an evaluated model, the reported numbers partly measure self-agreement.

assumptions (5)
  • ad hoc to paper GPT-4o-mini, prompted with the original dispute text and final decision, produces correct ground-truth labels for demand-wise decisions and argument evaluations.
    Section 5.1 creates gold labels this way with no human verification; this assumption is load-bearing for two of the three evaluation tasks.
  • ad hoc to paper The LLM-generated structured summaries, merged via majority vote, faithfully represent the disputes and the winning party.
    Section 4.1 constructs super summaries from three LLMs and takes the majority vote as the winning-party ground truth for D_AI.
  • domain assumption Rhetorical-role labels assigned by opennyai are a valid reference for evaluating structured-summary quality.
    Section 4.1 uses sentence-level rhetorical roles as pseudo-references; the paper notes these labels 'themselves may not be perfect.'
  • domain assumption Binary labels (stronger party, ACCEPTED/REJECTED, STRONG/WEAK) adequately represent judicial dispute outcomes.
    Both the system outputs and the ground truth reduce resolution to these categories (Sections 1, 4.1, 5.1); mixed or partial resolutions are not modeled.
  • domain assumption LLM pretraining provides sufficient world and legal knowledge for zero-shot dispute analysis.
    Section 4.2 expects models to rely on inherent world knowledge; no domain-specific training or retrieval is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DRAssist: Dispute Resolution Assistance using Large Language Models." pith.science (2026). https://pith.science/paper/MOJNAZWV

@misc{pith2026250901962,
  author       = {Pith},
  title        = {Pith review of: DRAssist: Dispute Resolution Assistance using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOJNAZWV}},
  note         = {Machine review of arXiv:2509.01962}
}
read the original abstract

Disputes between two parties occur in almost all domains such as taxation, insurance, banking, healthcare, etc. The disputes are generally resolved in a specific forum (e.g., consumer court) where facts are presented, points of disagreement are discussed, arguments as well as specific demands of the parties are heard, and finally a human judge resolves the dispute by often favouring one of the two parties. In this paper, we explore the use of large language models (LLMs) as assistants for the human judge to resolve such disputes, as part of our DRAssist system. We focus on disputes from two specific domains -- automobile insurance and domain name disputes. DRAssist identifies certain key structural elements (e.g., facts, aspects or disagreement, arguments) of the disputes and summarizes the unstructured dispute descriptions to produce a structured summary for each dispute. We then explore multiple prompting strategies with multiple LLMs for their ability to assist in resolving the disputes in these domains. In DRAssist, these LLMs are prompted to produce the resolution output at three different levels -- (i) identifying an overall stronger party in a dispute, (ii) decide whether each specific demand of each contesting party can be accepted or not, (iii) evaluate whether each argument by each contesting party is strong or weak. We evaluate the performance of LLMs on all these tasks by comparing them with relevant baselines using suitable evaluation metrics.

Figures

Figures reproduced from arXiv: 2509.01962 by the authors.

Figure 1
Figure 1. Workflow in dispute resolution assistance system. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Noun and verb word clouds for the datasets [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Relative prevalence of rhetorical roles in the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Evaluation of justifications generated by multiple [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages

  1. [1]

    Olga Alejandra Alcántara Francia, Miguel Nunez-del Prado, and Hugo Alatrista- Salas. 2022. Survey of text mining techniques applied to judicial decisions pre- diction. Applied Sciences 12, 20 (2022), 10200

    work page 2022

  2. [2]

    M. Butani. 2016. Tax Dispute Resolution-Challenges And Opportunities For India . Lexis-Nexis

    work page 2016

  3. [3]

    Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The Muppets straight out of Law School. In Findings of the ACL: EMNLP 2020 . 2898–2904

    work page 2020

  4. [4]

    and Langford M

    Behn D. and Langford M. 2017. Trumping the environment? An empirical perspective on the legitimacy of investment treaty arbitration. Journal of World Investment and Trade 18, 1 (2017), 14–61

    work page 2017

  5. [5]

    Lisa Ferro, John Aberdeen, Karl Branting, Craig Pfeifer, Alexander Yeh, and Amartya Chakraborty. 2019. Scalable methods for annotating legal-decision corpora. In Natural Legal Language Processing Workshop 2019 . 12–20

    work page 2019

  6. [6]

    Jianhua Guan, Zuguo Yu, Yongan Liao, Runbin Tang, Ming Duan, and Guosheng Han. 2024. Predicting Critical Path of Labor Dispute Resolution in Legal Domain by Machine Learning Models Based on SHapley Additive exPlanations and Soft Voting Strategy. Mathematics 12, 2 (2024), 272

    work page 2024

  7. [7]

    Roger Guimera and Marta Sales-Pardo. 2011. Justice Blocks and Predictability of U.S. Supreme Court Votes. PLoS One 6, 11 (2011), 1–8

    work page 2011

  8. [8]

    Prathamesh Kalamkar, Aman Tiwari, Astha Agarwal, Saurabh Karn, Smita Gupta, Vivek Raghavan, and Ashutosh Modi. 2022. Corpus for Automatic Structuring of Legal Documents. In LREC 2022. 4420–4429

    work page 2022

Show all 18 references
  1. [9]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213

    2022

  2. [10]

    William Massey, Giles Goodfellow, David Ewart, Rupert Baldry, and Andrew Hitchmough. 2024. Tax Litigation Handbook. Bloomsbury Professional

    2024

  3. [11]

    Medvedeva, M

    M. Medvedeva, M. Vols, and M. Wieling. 2020. Using machine learning to predict decisions of the European Court of Human Rights.AI and Law 28 (2020), 237–266

    2020

  4. [12]

    Rajat Mohan. 2016. Guide to Goods and Service Tax . Bharat Law

    2016

  5. [13]

    Matthew Renze and Erhan Guven. 2024. The effect of sampling temperature on problem solving in large language models. arXiv:2402.05201 (2024)

    2024 arXiv

  6. [14]

    Ruihao Shui, Yixin Cao, Xiang Wang, and Tat-Seng Chua. 2023. A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction. In Findings of the Association for Computational Linguistics: EMNLP 2023 . 7337–7348

    2023

  7. [15]

    David Tsurel, Michael Doron, Alexander Nus, Arnon Dagan, Ido Guy, and Dafna Shahaf. 2020. E-commerce dispute resolution prediction. In Proceedings of the 29th ACM CIKM 2020. 1465–1474

    2020

  8. [16]

    Wayan Oger Vihikan, Meladel Mistica, Inbar Levy, Andrew Christie, and Timothy Baldwin. 2021. Automatic resolution of domain name disputes. In Proceedings of the Natural Legal Language Processing Workshop 2021 . 228–238

    2021

  9. [17]

    Wei Yang, Luchen Tan, Chunwei Lu, Anqi Cui, Han Li, Xi Chen, Kun Xiong, Muzi Wang, Ming Li, Jian Pei, et al. 2019. Detecting customer complaint escalation with recurrent neural networks and manually-engineered features. InProceedings DRAssist: Dispute Resolution Assistance usi...

    2019

  10. [18]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.