{"id":"2c9e12b3-c9fb-4ce7-8443-5497e80cb194","arxiv_id":"2509.01962","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DRAssist shows that zero-shot LLMs with chain-of-thought prompting can outperform simple baselines at suggesting dispute outcomes, though the evaluation ground truth is itself generated by LLMs.","lead":"This paper builds DRAssist, a system that uses large language models to summarize car insurance and domain name disputes and to suggest which side is stronger, which demands should be accepted, and which arguments are strong. It finds that a chain-of-thought prompting strategy improves accuracy over direct prompting, but the evaluation relies on model-generated ground truth.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM-generated gold labels make the evaluation circular: GPT-4o-mini creates demand/argument ground truth and is also an evaluated model, so macro-F1 may reflect self-consistency rather than resolution quality.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the evaluation is circular for at least the demand-wise and argument-wise tasks, because GPT-4o-mini generates the gold labels and is also an evaluated system. I do not see a more fundamental flaw that would make the S3-vs-S2 comparison unsalvageable in principle; a human-annotated sample could resolve it. The paper explicitly says a user study is future work, and no significance tests are provided, but those are supporting weaknesses rather than a separate fatal objection. Thus the appropriate verdict remains CONDITIONAL, and my stress-test does not shift the reader's verdict.","tokens_in":16805,"tokens_out":5977,"duration_ms":68805,"concrete_test":"Take a random sample of ~25–30 disputes from each dataset and have two independent legal annotators (with a third adjudicator for disagreements) label: (a) official winning party, (b) accept/reject for each demand, and (c) strong/weak for each argument, working from the original ruling text. Compute human–LLM gold agreement and then recompute the Table 6 macro-F1 for S1/S2/S3 on this sample using only human labels. If human-gold agreement is low (<0.8) or if S3's advantage over S2/direct prompting shrinks or reverses, the reported F1s measure self-consistency rather than dispute-resolution quality; if S3 remains better on human labels, the circularity objection is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result—S3 (CoT) reaching macro-F1 0.78 vs. 0.53 random baseline for D_AI stronger-party prediction, and the claimed S3 > S2 > S1 ordering—rests on gold labels that are not independently grounded. In §5.1, demand-wise and argument-wise gold labels are created by prompting GPT-4o-mini with the original dispute text 'which contains the final decision,' asking it to infer whether each demand was accepted and whether each argument was viewed favourably by the judge. GPT-4o-mini is one of the three evaluated LLMs, so for these two tasks the reported accuracy/F1 measures agreement between GPT-4o-mini as annotator and GPT-4o-mini as predictor. For D_AI, even the stronger-party gold is not human-verified: §4.1 derives the winning party by majority vote among structured summaries produced by the same three LLM families from texts containing the final decision. The paper's own conclusion defers a human user study to future work. Consequently, the absolute numbers in Table 6 are not evidence that the predictions match real adjudication. The relative S3 advantage could also be inflated if S3's instruction to score argument strength aligns with the prompt used to generate argument gold, which asks the same model to infer judge favorability; no human agreement or significance testing is reported to rule this out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DRAssist, a prototype dispute-resolution assistance system that uses large language models in a zero-shot manner. It contributes two datasets (104 auto-insurance disputes from NCDRC, 351 domain-name disputes from WIPO), an LLM-based structured summary for each dispute, and three LLM prompting strategies S1 (direct stronger-party prediction), S2 (stronger party plus demand-wise decisions), and S3 (chain-of-thought: argument-strength evaluation followed by stronger-party and demand-wise predictions). The authors evaluate Mistral-7B, Llama-3-8B, and GPT-4o-mini, plus a majority-vote ensemble, reporting accuracy and macro-F1 for stronger-party prediction, demand-wise decisions, and argument-wise evaluation. The main claim is that S3 outperforms S2, which outperforms S1, and that S3 substantially beats majority/random baselines, e.g., macro-F1 0.78 vs. 0.53 for stronger-party prediction on the auto-insurance dataset. The paper also reports summarization quality via ROUGE/BERTScore against noisy rhetorical-role references and analyzes a few error cases.","tokens_in":17142,"tokens_out":4635,"duration_ms":53068,"significance":"If the reported results were grounded in human-validated ground truth, the paper would be a useful demonstration that zero-shot LLMs can provide multi-level assistance in dispute resolution, with the notable strengths of introducing two new domain datasets, releasing structured summaries and ground-truth annotations, and comparing several LLMs and prompting strategies. The error analysis in Section 5.6 is valuable and honest about failure modes. However, the central evaluation is currently not trustworthy for real adjudication: the demand-wise and argument-wise gold labels are generated by GPT-4o-mini itself, and the winning-party gold for the auto-insurance dataset is derived from LLM summaries. The reported F1 scores therefore largely measure self-consistency between an LLM annotator and LLM predictors from the same family. The paper explicitly defers a human user study to future work. Because the quantitative claims—especially the headline S3 advantage over S1/S2—rest on this circular evaluation, the contribution in its present form is a promising pipeline and dataset, not yet a validated demonstration of dispute-resolution assistance.","major_comments":[{"comment":"The demand-wise and argument-wise gold labels are created by prompting GPT-4o-mini with the original dispute text that contains the final decision, asking it to infer per-demand acceptance and per-argument judge favorability. GPT-4o-mini is also one of the three evaluated LLMs. Consequently, the reported demand/argument accuracies and macro-F1 values (e.g., D_AI argument-wise S3 Ensemble macro-F1 0.60, Table 6) measure agreement between GPT-4o-mini as annotator and GPT-4o-mini as predictor, not agreement with any human judge. This circularity can also inflate the relative advantage of S3, because the S3 prompt asks the model to label each argument STRONG/WEAK using criteria similar to those in the gold-generation prompt. To support the paper's claims, the authors need human-validated gold labels on a sample (or an independent set of human decisions) and should report agreement between LL","section":null},{"comment":"For D_AI, the winning-party gold standard is obtained by majority vote among three LLM-generated structured summaries (Mistral, Llama, GPT-4o-mini), with no human verification. The final decision text is included in the original dispute, but the summaries themselves may hallucinate or omit the winner. Since the headline result for stronger-party prediction (S3 Ensemble macro-F1 0.78 on D_AI) is computed against this LLM-derived gold, an incorrect winner in even a modest fraction of the 104 disputes would materially change the reported accuracy. The D_DN winning-party gold is extracted by regex from the WIPO decisions and is more reliable, but it is not validated either. The authors should verify the winning-party labels against the original decisions by human inspection or by extracting from the explicitly structured 'Decision' sections.","section":null},{"comment":"The paper claims that 'S3 is better than S2 and S2 is better than S1 for both the datasets,' but no significance testing or confidence intervals are reported. Several differences in Table 6 are within sampling noise: for D_AI demand-wise decisions, S2 Ensemble macro-F1 is 0.63 and S3 Ensemble is 0.62; for D_DN stronger-party prediction, S3 Ensemble macro-F1 is 0.52, which is worse than S3 Llama (0.62) and comparable to S2 GPT (0.53). Point estimates on 104 or 351 disputes cannot support a strict ordering, especially for differences of 0.01–0.03. The authors should report paired significance tests (e.g., McNemar's test or bootstrap confidence intervals) and effect sizes for the S3-vs-S2 and S2-vs-S1 comparisons.","section":null}],"minor_comments":[{"comment":"The phrase 'aspects or disagreement' should be 'aspects of disagreement' or 'disagreement aspects'.","section":"Abstract"},{"comment":"The summarization evaluation uses rhetorical-role labels from 'opennyai' as references; the paper acknowledges these labels are imperfect. Given the low ROUGE scores (e.g., R1-F1 0.30–0.42), the summarization quality claim should be framed as approximate and perhaps supplemented with a small human evaluation of sampled structured summaries.","section":"Section 4.1 and Table 3"},{"comment":"Justification quality is computed only for disputes where the stronger-party prediction is correct. This selection bias should be stated more prominently, as it may overstate justification quality on easier disputes and makes cross-strategy comparisons of justification metrics difficult to interpret.","section":"Section 5.2"},{"comment":"The ensemble mapping for demand/argument labels relies on one-to-one matching to the ground truth; errors in this matching propagate to the ensemble vote. The paper should state how ties in majority voting are broken and what happens when one LLM omits a demand/argument entirely.","section":"Section 5.4"},{"comment":"The cosine-similarity matching uses a linear sum assignment algorithm, but no similarity threshold is reported. Very low-similarity matches could create spurious alignments between the LLM output and ground truth; an analysis or threshold would strengthen the evaluation.","section":"Section 5.1"},{"comment":"The random baseline macro-F1 values are not 0.5 in several rows (e.g., 0.53 for D_AI stronger-party). Since random labels are class-dependent, state the random-label generation procedure and report variance over multiple runs.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a systems/application paper whose novelty depends on the new datasets and the multi-level assistance pipeline. The central quantitative claims are currently undermined by LLM-generated gold labels, so the revision needs either a human-validated gold set or a clearly framed re-analysis as a self-consistency study. The authors also promise to release the datasets upon acceptance; this is important for reproducibility and should be confirmed. The paper's fit with a CL/legal-NLP venue is reasonable if the evaluation is strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: DRAssist is a readable, honestly-scoped prototype for using LLMs to assist human adjudicators. The new datasets (104 NCDRC auto-insurance disputes, 351 WIPO domain name disputes) and the three-level output (stronger party, demand-wise, argument-wise) are genuinely useful scaffolding for legaltech assistance. The paper also does something right that many LLM papers skip: it includes concrete error analysis showing how structured-summarization errors propagate and where reasoning diverges across models.\n\nThe weakness is the evaluation's circularity, and it is not minor. Demand-wise and argument-wise gold labels come from prompting GPT-4o-mini with the original dispute text that contains the final decision. GPT-4o-mini is also one of the three evaluated models. For those two tasks, the reported accuracy/F1 mostly measures how often GPT-4o-mini agrees with itself. For the stronger-party task in D_AI, the gold label is a majority vote among structured summaries produced by the same three LLMs from texts containing the final decision—again, no human verification. So Table 6's absolute numbers are not evidence that predictions match real adjudication. The S3-over-S1/S2 ordering is plausible, but it could be inflated if S3's argument-strength instruction aligns with the prompt used to generate the argument-wise gold, asking the same model to infer judge favorability. No human agreement, no significance tests, and no supervised baselines make that concern hard to dismiss.\n\nThe paper itself is not hiding this: it says a user study is future work, and the datasets are only promised 'upon acceptance.' That is honest, but it means the contributions are, right now, a task formulation and two datasets that are not yet public.\n\nProportionately: if you need a proof that CoT beats direct prompting for legal-style tasks, this paper doesn't get you there. If you want a concrete instance of the LLM-as-annotator trap, this is a good case study. I'd send it to review, but with a clear message that the evaluation needs human gold labels or at least a human-verified subset, and the datasets need release. The writing and framing deserve referee time.","headline":"A cleanly written prototype for LLM-assisted dispute resolution with two new datasets, but the headline CoT result is undercut by gold labels generated by the same model family being tested.","tokens_in":17606,"tokens_out":1680,"would_cite":false,"duration_ms":18496,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Asking an LLM to grade each argument strong or weak before naming the stronger party and ruling on demands markedly improves dispute assistance, lifting macro-F1 from 0.53 to 0.78 on automobile-insurance disputes.","keywords":["dispute resolution","large language models","chain-of-thought prompting","zero-shot reasoning","argument strength evaluation","structured summarization","insurance disputes","domain name disputes"],"falsifier":"Have adjudicators — retired consumer-forum members for the insurance disputes, trademark and arbitration lawyers for the domain-name disputes — label the winning party, each demand, and each argument as strong or weak on a random sample of 50–100 disputes per dataset, then recompute accuracy and macro-F1 for S1, S2, and S3 against those human labels. If the S3 advantage over S2 and S1 collapses, or if S3 matches the LLM-generated gold labels while disagreeing with the human ones, the central claim is refuted. Have the same humans rate the arguments separately to locate where disagreement origi","tokens_in":16684,"feed_emoji":"⚖️","tokens_out":23542,"duration_ms":211770,"temperature":0.7,"pith_summary":"This paper claims that large language models can genuinely help a human judge or arbitrator prepare a dispute resolution, provided the model works through the dispute in the right order. DRAssist first converts an unstructured dispute text into a fixed structure — agreed facts, points of disagreement, each side's arguments and demands — and then prompts an LLM with three strategies; the finding is that the chain-of-thought strategy S3 (score each argument strong or weak first, then name the overall stronger party and rule each demand) beats the two simpler strategies on both test domains. On automobile-insurance disputes, the S3 ensemble reaches macro-F1 — the balanced average of precision and recall across the two outcome classes — of 0.78, against 0.53 for the best baseline; on domain-name disputes the best S3 model reaches 0.62 versus 0.46, with argument-wise evaluation at 0.73. The paper frames this deliberately as assistance, not automation: the output is an explained first-cut recommendation for a human decision-maker, aimed at reducing the effort, time, and subjectivity in high-volume pre-legal forums. If the claim holds, useful adjudication support is available without training on any corpus of past disputes.","feed_headline":"Grading each argument first lifts dispute AI to 0.78 macro-F1","feed_subtitle":"Asking an LLM to grade each argument before ruling beats direct prompts on two dispute datasets","key_machinery":"The load-bearing object is the S3 prompt, a zero-shot chain-of-thought template: it instructs the model to (1) rewrite and label every argument of both parties as STRONG or WEAK, (2) identify the overall stronger party from those evaluations, and (3) label each demand ACCEPTED or REJECTED, with a consistency rule barring opposing demands from sharing a label. The two weaker strategies — S1 (stronger party only) and S2 (demand rulings without argument scoring) — isolate the contribution of the intermediate argument-evaluation step. The other load-bearing piece is the structured summary: each dispute is converted, element by element (agreed facts, disagreement aspects, arguments, demands, prio","core_discovery":"The paper's central claim: the chain-of-thought prompting strategy, S3, gives the best dispute-resolution assistance of the three strategies tested. To use S3, the LLM first labels each argument of each party STRONG or WEAK, then names the overall stronger party and rules each demand ACCEPTED or REJECTED, barring conflicting demands from the same label. For stronger-party prediction, the ordering S3 > S2 > S1 holds on both datasets: macro-F1 0.78 vs 0.53 baseline on automobile-insurance disputes, and 0.62 vs 0.46 on domain-name disputes; under S3, argument evaluation reaches 0.73 and demand rulings 0.64. A further result: despite predicting better, S3's justifications score lower on ROUGE an","pith_inferences":["Since the gold labels for demands and arguments are produced by prompting GPT-4o-mini on text that already contains the final decision, and the winning-party gold standard is a majority of LLM summaries that include the same model family, the reported F1 scores are best read as measures of model self-consistency; they become statements about agreement with human adjudication only after re-scoring ","If the S3 advantage survives human labels, the likely reason is that structuring the question — not any domain tuning — drives the gain, so the same template should transfer to other adjudicative text domains such as tax, banking, or health-insurance complaints; testing this on one additional corpus would be a cheap validation.","The strong complainant bias observed in the domain-name data (recall for the respondent as low as 0.31 even in the best setting) suggests the models lean on a pretraining prior — complainants usually win in such disputes — rather than case content; a party-label swap (keep arguments, exchange 'complainant' and 'respondent' names) would separate label-prior from content-based reasoning.","The paper leaves open whether the structured summary is what carries the accuracy: running the same S3 prompt directly on the raw dispute text would isolate the contribution of the summarization scaffold, and the error analysis predicts accuracy would drop if summaries are imperfect."],"forward_implications":["Useful first-cut adjudication support becomes available with zero supervision: no corpus of past cases is needed for training, which matters for high-volume pre-legal forums where labeled data is scarce.","The consistency constraint in the S3 prompt (conflicting demands cannot both be ACCEPTED) shows that structurally sound resolutions can be coaxed from an LLM by prompt design rather than by post-hoc repair.","Argument-wise STRONG/WEAK ratings are a usable byproduct: parties could receive a per-argument strength breakdown with justifications, independent of the ultimate verdict.","Because S3 predicts better but justifies worse than S1/S2, a practical system would run S3 for the prediction and a separate direct prompt for the narrative rationale, trading a little accuracy for readable explanations.","The paper's error analysis implies the summarization stage sets a ceiling on resolution accuracy — errors in the structured summary propagate into wrong verdicts — so improving the upstream summary is a direct lever on downstream performance."],"supporting_citations":[{"why":"Supplies the zero-shot chain-of-thought prompting principle on which the best-performing S3 strategy is built, and that the paper credits for the improvement.","marker":"[9]"},{"why":"Defines the domain-name dispute prediction task and decision corpus that the paper follows in scraping its own dataset of 351 cases with formal respondent arguments.","marker":"[16]"},{"why":"Provides the rhetorical-role classifier and taxonomy used to estimate the quality of the structured summaries and to profile the two datasets.","marker":"[8]"},{"why":"Supports setting the sampling temperature to 0 for problem-solving runs, the configuration behind the reproducibility of all reported numbers.","marker":"[13]"},{"why":"Defines BERTScore, one of the three metrics used to compare LLM-generated justifications against the gold-standard justifications.","marker":"[18]"}],"fun_headline_variants":["Grade each argument first: dispute AI reaches 0.78 macro-F1","Chain-of-thought prompting lifts dispute-resolution AI to 0.78 F1","For dispute AI, per-argument grading beats direct prompts","LLM assistant for judges: grade arguments before ruling","S3 strategy in DRAssist: argue-then-rule improves decisions"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The labels that the evaluation scores the models against are produced by the same family of models being tested: GPT-4o-mini is prompted with the original dispute text — which already contains the final decision — to mark each demand and argument, and the winning-party gold standard is a majority vote across LLM-generated summaries that include GPT-4o-mini (Section 5.1); if those labels diverge from what human adjudicators would decide, the reported accuracies measure self-co","fun_headline_variants_meta":{"raw":{"variants":["Grade each argument first: dispute AI reaches 0.78 macro-F1","Chain-of-thought prompting lifts dispute-resolution AI to 0.78 F1","For dispute AI, per-argument grading beats direct prompts","LLM assistant for judges: grade arguments before ruling","S3 strategy in DRAssist: argue-then-rule improves decisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1499,"prompt_tokens":817,"completion_tokens":682,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":588}},"tokens_in":561,"tokens_out":682,"duration_ms":7438,"temperature":1.0,"reasoning_tokens":588,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:00:59.611548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have adjudicators — retired consumer-forum members for the insurance disputes, trademark and arbitration lawyers for the domain-name disputes — label the winning party, each demand, and each argument as strong or weak on a random sample of 50–100 disputes per dataset, then recompute accuracy and macro-F1 for S1, S2, and S3 against those human labels. If the S3 advantage over S2 and S1 collapses, or if S3 matches the LLM-generated gold labels while disagreeing with the human ones, the central claim is refuted. Have the same humans rate the arguments separately to locate where disagreement origi","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the domain-name dispute prediction task and decision corpus that the paper follows in scraping its own dataset of 351 cases with formal respondent arguments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the rhetorical-role classifier and taxonomy used to estimate the quality of the structured summaries and to profile the two datasets."}],"review_version":1}