SAMVAD: A Multi-Agent System for Simulating Judicial Deliberation Dynamics in India

Omkaar Jayadev Shetty; Pooja Agarwal; Prathamesh Devadiga

REVIEW 3 major objections 5 minor 12 references

SAMVAD: A Multi-Agent System for Simulating Judicial Deliberation Dynamics in India

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that grounding judge and counsel agents in a retrieval-augmented Indian legal knowledge base roughly doubles their legal-grounding scores, raises agreement, and improves verdict consistency across four open LLMs.

desk verdict A useful system paper with a clear architecture, but its headline RAG benefit is measured with a keyword-frequency metric that RAG itself inflates. read the letter →

arxiv 2509.03793 v1 pith:5ZPTYZ36 submitted 2025-09-04 cs.MA cs.AI

Prathamesh Devadiga , Omkaar Jayadev Shetty , Pooja Agarwal This is my paper

classification cs.MAcs.AI

keywords Multi-AgentSystemsAgent-BasedSimulationLegalAIJudicialDeliberationRetrieval-AugmentedGeneration(RAG)LLMAgentsIndianLawVerdictConsistency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SAMVAD is an LLM-powered multi-agent system that simulates Indian judicial deliberation with a Judge, Prosecution Counsel, Defense Counsel, and a bench of Adjudicators. The paper's central claim is that connecting the Judge and Counsel agents to a retrieval-augmented knowledge base of Indian legal texts markedly improves deliberation quality: across four open-weight LLMs, enabling RAG roughly doubles the Argument Grounding Score, increases agreement ratios and meaningful statements, and raises verdict consistency. If this is right, simulations of judicial reasoning can become more faithful and transparent by making agents cite actual legal sources, offering a way to study judicial panel dynamics without the ethical and practical barriers of real-court experiments.

What carries the argument

The load-bearing component is a Retrieval-Augmented Generation (RAG) pipeline over a vector knowledge base built from the Indian Penal Code, the Constitution of India, and the Code of Criminal Procedure. Legal texts are split into chunks, embedded, and stored in a vector database; when the Judge or a Counsel generates text, the query is embedded and similar legal chunks are retrieved into the prompt, allowing the agent to cite source documents. The Orchestrator then runs iterative deliberation rounds among the Adjudicators until at least eighty percent consensus or a round limit is reached. The paper's principal quantitative instrument is the Argument Grounding Score, defined as the frequenc

What would settle it

Blind legal review: have Indian lawyers or judges score the RAG and non-RAG transcripts from the same case files and models for legal soundness without knowing which condition produced them. If RAG arguments do not score higher despite roughly doubling keyword frequency, the paper's central empirical claim is refuted. A second check: run SAMVAD on cases with known real-world Indian court outcomes; if RAG does not improve verdict accuracy, its benefit is stylistic rather than substantive.

Watch

Extended reading notes

Core claim

The paper's central empirical discovery is that turning on retrieval-augmented generation for the Judge and Counsel agents changes deliberation quality more than swapping the underlying language model does. Across all four tested open-weight models, RAG roughly doubled the Argument Grounding Score (for one model, from 0.21 to 0.42), raised average meaningful statements per adjudicator, lifted final agreement ratios, and moved verdict consistency from Low or Medium to High or Very High. The authors interpret this as evidence that grounding agent reasoning in an external, citable legal knowledge base is critical for achieving high-quality, consistent, and explainable simulation outcomes.

Load-bearing premise

The load-bearing premise is that the Argument Grounding Score, defined as the frequency of predefined case keywords in an agent's statement, measures the legal soundness of that statement; if keyword frequency does not track legal quality, the quantitative evidence that RAG helps collapses.

Editorial extensions

If this is right

If the central claim holds, simulating judicial deliberation no longer requires pretending the LLM already knows the law: agents can retrieve and cite actual statutes, making each simulated argument auditable against source documents.
The consistency result implies that running the same case multiple times with RAG yields the same verdict far more often than without RAG, a prerequisite for using such simulations as reproducible testbeds in legal AI.
A configurable judge, counsel, and bench simulation gives researchers a way to study group deliberation, such as the effect of evidence presentation or panel composition on verdicts, without ethical constraints on real judicial experiments.
The RAG benefit appears model-agnostic across the four tested open-weight LLMs, suggesting the grounding gain is not an artifact of one model.
The citation mechanism ties each legal instruction and argument to a source document, supporting explainability of AI-generated legal reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, RAG is enabled only for Judge and Counsel agents, so a natural testable extension is to give Adjudicators retrieval access too and measure whether grounding scores and consensus quality rise further.
Because the grounding score counts keyword frequency, part of the observed doubling may reflect the RAG prompt inserting legal vocabulary rather than improving underlying reasoning; a semantic or expert-rated metric is needed before treating RAG as validated.
The architecture should transfer to other codified legal systems by swapping the corpus, but in precedent-heavy common-law jurisdictions the retrieval unit would need case law and an explicit model of ratio decidendi, which statute-chunk retrieval does not provide.
The five-case evaluation is too small to establish general claims about deliberation dynamics; a corpus of hundreds of varied cases would test whether the observed one-to-two-round consensus and high agreement ratios persist.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

SAMVAD is a clean, readable system paper about using LLM agents to simulate Indian judicial deliberation. The new bit is the integration: judge and counsel agents are augmented with RAG over a vector store built from the IPC, the Constitution, and the CrPC, and they emit citations to retrieved text. The components are all known, but the particular combination is not something I've seen in the prior work they cite. The architecture is modular and the orchestration is straightforward. If you want a template for a legal-role-play MAS, this is a good starting point.

The evaluation is the soft spot. The headline claim is that RAG roughly doubles the 'Argument Grounding Score.' That score is defined in Sec 5.1 as the frequency of predefined case keywords appearing in agent statements. The stress-test note is right: the case files already include those keywords (Sec 3.1), and the RAG-enabled agents receive retrieved chunks of statutes that are full of legal vocabulary, so the conditioning naturally pushes the keyword count up. The no-RAG condition lacks that lexical priming. So the doubling is very plausibly a measure of lexical exposure, not legal quality. There is no expert annotation or validation of the metric. On top of that, the ablation in Table 2 reports one run per condition, with no error bars or significance tests, and the 'consistency' labels (Low/Medium/High) are not defined with any counts. The five sample cases in Table 1 are unrandomized and small.

That said, the architecture itself holds up. The agreement ratios and the increase in meaningful statements are directionally sensible, even if not rigorously established. The citation mechanism is a nice touch for explainability. The related work is appropriately cited, and I don't see a circularity problem beyond the metric issue.

This paper is for researchers building legal simulations or studying multi-agent deliberation; it is not evidence about real judicial practice. As is, I would not cite its empirical claims, but I'd cite it as an example of an applied LLM-MAS system. It deserves a serious referee, with the expectation of major revisions: either replace the grounding metric with expert judgment or at least validate the keyword proxy, run multiple seeds, report variance, and release code/data. If the authors do that, the system would be a solid contribution.

Referee Report

3 major / 5 minor

Summary. The paper presents SAMVAD, a multi-agent LLM system that simulates judicial deliberation in the Indian legal context. The architecture includes a Judge agent, Prosecution and Defense Counsel agents, and multiple Adjudicator agents, coordinated by an Orchestrator. A central design choice is Retrieval-Augmented Generation (RAG) over a vectorized knowledge base built from the Indian Penal Code, the Constitution of India, and the Code of Criminal Procedure; the Judge and Counsel agents use this RAG pipeline to produce citable legal instructions and arguments. The Adjudicator agents deliberate over multiple rounds until a consensus threshold is reached. The evaluation consists of five illustrative case runs and an ablation study across four open-source LLMs with and without RAG. The paper's main empirical claim, stated in Section 5.3, is that enabling RAG leads to higher agreement ratios, roughly doubled argument grounding scores, more meaningful statements, and higher verdict consistency.

Significance. If the empirical claims were established, SAMVAD would be a useful contribution: it provides a modular, configurable MAS framework for studying legal deliberation, and it is one of the first attempts to integrate a domain-specific RAG pipeline into an LLM multi-agent judicial simulation. The architectural design is coherent and the choice of grounding in Indian legal texts is well motivated. However, the paper's central quantitative evidence for the RAG benefit is currently weak: the Argument Grounding Score is calculated as keyword frequency, the evaluation reports no error bars or statistical tests, and no external validation against expert legal judgment is provided. The contribution is therefore best characterized as a promising system description with preliminary, but not yet conclusive, empirical support.

major comments (3)

[§5.1, Table 2] The Argument Grounding Score is defined as the frequency of predefined case keywords appearing in agent statements. Because the case file already contains those keywords (Section 3.1) and RAG-enabled agents receive retrieved chunks from IPC/Constitution/CrPC that naturally contain the same legal vocabulary, the RAG-vs-no-RAG comparison in Table 2 is confounded: the roughly doubling of the score (e.g., 0.21 to 0.42 for Qwen-2.5-7B) may measure lexical exposure to retrieved legal text rather than the legal soundness of the reasoning. Moreover, each configuration in Table 2 appears to correspond to a single run: there are no error bars, no multiple seeds, and no significance tests. This is load-bearing because Section 5.3 concludes that 'enabling RAG led to higher agreement ratios, significantly improved argument grounding scores (roughly doubling them), more meaningful statements, and high
[§5.2, §5.1] Table 1 reports one run per case for five cases, with no case-selection criteria, no randomization, and no ground-truth verdicts or expert comparisons. The consistency measure defined in Section 5.1 ('percentage of runs yielding the most common verdict') cannot be estimated from a single run, yet Table 2 reports qualitative consistency labels ('Very High', 'Medium', 'Low') without stating the number of runs, thresholds, or distributions. Claims about 'simulation fidelity' and verdict consistency are therefore unsupported by the reported evidence. The authors should provide multiple independent runs per configuration, report the distribution of verdicts, and ideally compare the simulated outcomes against a gold standard or human evaluation.
[§3.2, §5.3] The paper highlights source citations as a key explainability mechanism, stating in Section 3.2 that RAG-enabled agents generate 'legally precise, complete with citations to the source material.' However, no evaluation of citation accuracy, citation grounding, or whether the retrieved legal text actually supports the agents' assertions is presented. Table 2 does not include any citation-related metric. Similarly, the 'Avg. Stmts.' and 'meaningful statements' metrics are defined only by a vague criterion ('well-supported justifications') in Section 5.1, without a rubric or inter-annotator agreement. Since these measures are part of the central RAG-improvement claim, the authors should either provide a concrete definition and validation or remove them from the list of claimed benefits.

minor comments (5)

[Title/Abstract] The system name is written inconsistently: 'SAMVAD' in the title and abstract, but 'SAMV AD' in the running header, abstract, and body. Please standardize.
[§4, Step 4] The consensus threshold appears as '¿80% agreement' in the PDF text; this is an encoding artifact and should read '>80%'. Please fix the source encoding.
[§5.3, Table 2] The table header uses 'Ground Score' instead of the full term 'Argument Grounding Score' used in Section 5.1. Also, the qualitative consistency labels ('Very High', 'High', 'Medium', 'Low') are not defined anywhere; please specify the underlying quantitative measure.
[§2, Reference [1]] The Related Work section refers to 'SimuCourt, demonstrating the potential of agent-based systems,' but Reference [1] is titled 'AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation.' Please check whether the citation and text match.
[§5] Section 5 begins with 'a comprehensive evaluation plan,' but the paper actually reports completed simulation results. Please clarify which parts are planned and which are executed, and include details such as number of seeds, temperature settings, prompt versions, and computational environment to support reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

The RAG benefit claim rests on a keyword-frequency metric that mirrors the keywords supplied by the RAG pipeline, making the reported doubling partly an artifact of the metric's definition.

self definitional [Section 5.1 (Evaluation Metrics), Section 3.1 (Architecture Design), compared with Section 5.3 (Ablation Studies)]
"Argument Grounding Score: ... This is calculated by checking the frequency of predefined case keywords in agent statements. ... Each case file includes ... a set of keywords for grounding the analysis. ... Across all models, enabling RAG led to ... significantly improved argument grounding scores (roughly doubling them) ..."

The paper's main quantitative evidence for RAG's benefit is the Argument Grounding Score, defined as the frequency of case keywords in agent statements. Those keywords are part of the case-file input, and RAG supplies the Judge and Counsel agents with retrieved chunks of the IPC/Constitution/CrPC, legal texts that necessarily contain the same vocabulary. Hence the metric's numerator is the presence of the very tokens the intervention injects; the no-RAG condition lacks this lexical priming. The Table 2 doubling (e.g., Qwen 0.21 to 0.42) therefore largely measures the fact that keyword-rich legal text was added to the prompt, not an independent, externally validated improvement in legal reasoning quality. The claim 'RAG improves grounding' reduces by construction to the way grounding is com

full rationale

There is no fitted parameter or self-derived derivation chain in the paper besides the evaluation metric. The central claim that RAG roughly doubles argument grounding is circular in a narrow but important sense: the metric counts the same keywords that the RAG pipeline's retrieved legal chunks supply. That makes the reported improvement a mechanical consequence of the experimental manipulation rather than an independent validation of legal soundness. The other metrics (agreement ratio, meaningful statements, consistency) are either under-specified or not obviously circular. No self-citations are load-bearing; the cited prior work is external. Verdict consistency is also not defined in a way that would reduce to inputs. The paper's architectural contribution (MAS + RAG + citations) is independent, but its headline quantitative result is weakened by the self-definitional metric, so the circularity score is 6.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper's conclusions depend on several choices not independently validated: the consensus threshold and keyword sets are chosen by the authors, and the central metric assumes keyword frequency equals legal grounding. The axioms list the unproved premises that the empirical claims rely on.

free parameters (3)

consensus_agreement_threshold = 0.80
Chosen by hand; the Orchestrator declares consensus at over 80% agreement, used in all reported runs.
maximum_deliberation_rounds = not stated in text
The workflow says 'up to a maximum limit' but the limit is never given; it shapes whether cases conclude as hung panels.
case keyword sets for Argument Grounding Score = per-case values, not listed
The grounding score counts occurrences of predefined case keywords in statements; these sets are authored by the researchers and are the only thing the score measures.

assumptions (3)

domain assumption LLM-agent deliberation is a meaningful proxy for human judicial deliberation
The entire simulation is presented as a study of judicial deliberation dynamics, but no validation against real judges or cases is provided (Sections 1, 4).
domain assumption The RAG knowledge base (Constitution, IPC, CrPC) is sufficient for the case set
All agent grounding is assumed to be complete enough for these five sample cases; no legal expert review is reported (Section 3.4).
ad hoc to paper Keyword frequency measures argument quality and legal grounding
The Argument Grounding Score (Section 5.1) equates grounding with counting predefined keywords; this metric is introduced by the authors for this paper and has no external validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SAMVAD: A Multi-Agent System for Simulating Judicial Deliberation Dynamics in India." pith.science (2026). https://pith.science/paper/5ZPTYZ36

@misc{pith2026250903793,
  author       = {Pith},
  title        = {Pith review of: SAMVAD: A Multi-Agent System for Simulating Judicial Deliberation Dynamics in India},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ZPTYZ36}},
  note         = {Machine review of arXiv:2509.03793}
}

read the original abstract

Understanding the complexities of judicial deliberation is crucial for assessing the efficacy and fairness of a justice system. However, empirical studies of judicial panels are constrained by significant ethical and practical barriers. This paper introduces SAMVAD, an innovative Multi-Agent System (MAS) designed to simulate the deliberation process within the framework of the Indian justice system. Our system comprises agents representing key judicial roles: a Judge, a Prosecution Counsel, a Defense Counsel, and multiple Adjudicators (simulating a judicial bench), all powered by large language models (LLMs). A primary contribution of this work is the integration of Retrieval-Augmented Generation (RAG), grounded in a domain-specific knowledge base of landmark Indian legal documents, including the Indian Penal Code and the Constitution of India. This RAG functionality enables the Judge and Counsel agents to generate legally sound instructions and arguments, complete with source citations, thereby enhancing both the fidelity and transparency of the simulation. The Adjudicator agents engage in iterative deliberation rounds, processing case facts, legal instructions, and arguments to reach a consensus-based verdict. We detail the system architecture, agent communication protocols, the RAG pipeline, the simulation workflow, and a comprehensive evaluation plan designed to assess performance, deliberation quality, and outcome consistency. This work provides a configurable and explainable MAS platform for exploring legal reasoning and group decision-making dynamics in judicial simulations, specifically tailored to the Indian legal context and augmented with verifiable legal grounding via RAG.

Figures

Figures reproduced from arXiv: 2509.03793 by the authors.

**Figure 1.** High-level architecture of the SAMVAD system. The Orchestrator manages the interaction between agents (Judge, Counsel, Adjudicators), who leverage a RAG pipeline connected to a legal knowledge base to process case files and produce a final judgment. 3.2 Agents Adjudicator Agents. These agents form the core of the deliberative body, simulating a judicial bench. Each Adjudicator agent operates autonomously, analyzing … view at source ↗

**Figure 2.** Detailed Simulation Workflow, illustrating the trial preparation and iterative jury deliberation phases [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[1]

arXiv 10 P

He, Z., Cao, P., Wang, C., et al.: AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation. arXiv 10 P. Devadiga et al. preprint arXiv:2403.02959 (2024)

arXiv 2024
[2]

Inferring Dynamic Networks from Marginals with Iterative Proportional Fitting

Jiang, C., Yang, X.: Agents on the Bench: Large Language Model Based Multi Agent Framework for Trustworthy Digital Justice. arXiv preprint arXiv:2402.18697 (2024)

work page Pith review arXiv 2024
[3]

arXiv preprint arXiv:2408.10343 (2024)

Pipitone, N., Alami, G.H.: LegalBench-RAG: A Benchmark for Retrieval- Augmented Generation in the Legal Domain. arXiv preprint arXiv:2408.10343 (2024)

arXiv 2024
[4]

IEEE Transactions on Cognitive and Developmental Systems, 10(3), 514– 529 (2018)

Rizk, Y., Awad, M., Tunstel, E.W.: Decision making in multiagent systems: A survey. IEEE Transactions on Cognitive and Developmental Systems, 10(3), 514– 529 (2018)

work page 2018
[5]

In: Proceedings of the 2019 International Conference on Machine Learning and Cy- bernetics (ICMLC), pp

Wan, X., Jin, P.: Judicial Reasoning Model Based on Multi-agent System. In: Proceedings of the 2019 International Conference on Machine Learning and Cy- bernetics (ICMLC), pp. 1-6. IEEE (2019)

work page 2019
[6]

Duke Law & Technology Review, 23(1), 1–45 (2023)

Grossman, M.R., Grimm, P.W., Brown, D.G., Xu, M.: The GPTJudge: Justice in a Generative AI World. Duke Law & Technology Review, 23(1), 1–45 (2023)

work page 2023
[7]

arXiv preprint arXiv:2301.05327 (2023)

Hamilton, S.: Blind Judgement: Agent-Based Supreme Court Modelling with GPT. arXiv preprint arXiv:2301.05327 (2023)

arXiv 2023
[8]

In: Proceedings of the AI for Good Governance (AIGOV) Workshop, CEUR Workshop Proceedings, vol

Casanovas, P., Hashmi, M., Poblet, M.: Generative AI and the Rule of Law. In: Proceedings of the AI for Good Governance (AIGOV) Workshop, CEUR Workshop Proceedings, vol. 3271 (2022)

work page 2022

Show all 12 references

[9]

Journal of Legal Analysis, 16(1), 235–262 (2024)

Liu, J.Z., Li, X.: How Do Judges Use Large Language Models? Evidence from Shenzhen. Journal of Legal Analysis, 16(1), 235–262 (2024)

2024
[10]

Bloomsbury Academic (2024)

Van der Sloot, B.: Regulating the Synthetic Society: Generative AI, Legal Ques- tions, and Societal Challenges. Bloomsbury Academic (2024)

2024
[11]

Loyola of Los Angeles Law Review, 57 (2024)

Chien, C.V., Kim, M., Raj, A., Rathish, R.: How generative AI can help address the access to justice gap through the courts. Loyola of Los Angeles Law Review, 57 (2024)

2024
[12]

International Journal for Court Administration, 15(1), 1 (2024)

Contini, F.: Unboxing generative AI for the legal professions: functions, impacts and governance. International Journal for Court Administration, 15(1), 1 (2024)

2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

arXiv 10 P

He, Z., Cao, P., Wang, C., et al.: AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation. arXiv 10 P. Devadiga et al. preprint arXiv:2403.02959 (2024)

arXiv 2024

[2] [2]

Inferring Dynamic Networks from Marginals with Iterative Proportional Fitting

Jiang, C., Yang, X.: Agents on the Bench: Large Language Model Based Multi Agent Framework for Trustworthy Digital Justice. arXiv preprint arXiv:2402.18697 (2024)

work page Pith review arXiv 2024

[3] [3]

arXiv preprint arXiv:2408.10343 (2024)

Pipitone, N., Alami, G.H.: LegalBench-RAG: A Benchmark for Retrieval- Augmented Generation in the Legal Domain. arXiv preprint arXiv:2408.10343 (2024)

arXiv 2024

[4] [4]

IEEE Transactions on Cognitive and Developmental Systems, 10(3), 514– 529 (2018)

Rizk, Y., Awad, M., Tunstel, E.W.: Decision making in multiagent systems: A survey. IEEE Transactions on Cognitive and Developmental Systems, 10(3), 514– 529 (2018)

work page 2018

[5] [5]

In: Proceedings of the 2019 International Conference on Machine Learning and Cy- bernetics (ICMLC), pp

Wan, X., Jin, P.: Judicial Reasoning Model Based on Multi-agent System. In: Proceedings of the 2019 International Conference on Machine Learning and Cy- bernetics (ICMLC), pp. 1-6. IEEE (2019)

work page 2019

[6] [6]

Duke Law & Technology Review, 23(1), 1–45 (2023)

Grossman, M.R., Grimm, P.W., Brown, D.G., Xu, M.: The GPTJudge: Justice in a Generative AI World. Duke Law & Technology Review, 23(1), 1–45 (2023)

work page 2023

[7] [7]

arXiv preprint arXiv:2301.05327 (2023)

Hamilton, S.: Blind Judgement: Agent-Based Supreme Court Modelling with GPT. arXiv preprint arXiv:2301.05327 (2023)

arXiv 2023

[8] [8]

In: Proceedings of the AI for Good Governance (AIGOV) Workshop, CEUR Workshop Proceedings, vol

Casanovas, P., Hashmi, M., Poblet, M.: Generative AI and the Rule of Law. In: Proceedings of the AI for Good Governance (AIGOV) Workshop, CEUR Workshop Proceedings, vol. 3271 (2022)

work page 2022