JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledge

Fumihito Nishino; Hiroaki Yamada; Ken Satoh; Nguyen Ha Thanh; Yusuke Miyao; Zhihan Cao

REVIEW 2 major objections 6 minor 14 references

A new dataset from the Japanese bar exam shows AI models know constitutional law better than civil or penal law.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 19:38 UTC pith:KSORVRUL

load-bearing objection Useful new Japanese legal benchmark; subject-difficulty claim needs per-subject label-balance checks. the 2 major comments →

arxiv 2511.22869 v1 pith:KSORVRUL submitted 2025-11-28 cs.CL

JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledge

Zhihan Cao , Fumihito Nishino , Hiroaki Yamada , Nguyen Ha Thanh , Yusuke Miyao , Ken Satoh This is my paper

classification cs.CL

keywords Japanese bar examlegal QALLM evaluationbenchmark datasettrue/false classificationCivil CodePenal CodeConstitution

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces JBE-QA, a dataset built from the multiple-choice section of the Japanese bar exam from 2015 to 2024. The authors turn each exam question into independent true/false statements, yielding 3,464 balanced items that cover the Civil Code, the Penal Code, and the Constitution. They then evaluate 26 large language models in zero-shot and four-shot settings. The main findings are that proprietary models with reasoning enabled score highest, that the Constitution is notably easier than the other two subjects, and that even the best model can be tripped up by a Penal Code question whose answer depends on precedent-based interpretation. The value of the dataset is that it gives the field a single, publicly derived yardstick for measuring Japanese legal knowledge in AI systems.

Core claim

JBE-QA is a benchmark derived from the multiple-choice section of the Japanese bar exam (2015–2024). Each official question is decomposed into independent true/false claims, producing 3,464 balanced binary items across the Civil Code, the Penal Code, and the Constitution. On this benchmark the best systems are proprietary language models with reasoning enabled, reaching F1 scores around 0.86 (a standard balance of precision and recall) in a four-shot setting, while open-weight and Japanese-specialised models generally trail. The paper reports a consistent subject ordering: the Constitution is easier than the Civil Code and the Penal Code for nearly every model. A case study involving the Pen

What carries the argument

The load-bearing device is the conversion of the original examination format—where each question is a set of statements and the examinee picks the correct combination of true/false values—into isolated binary classification instances. Each statement becomes an item with structured fields: subject, theme, instruction, question, lead-in context, and remarks. This decomposition is what makes automated evaluation tractable and fine-grained, since a model's knowledge can be scored per legal claim rather than per option combination. The authors also introduce a faithfulness score that records whether a model follows the instruction to output only a 0 or 1, which matters because some models append

Load-bearing premise

The dataset's gold labels are valid only if every statement's truth can be decided independently, exactly as the official answer key marks it; if any statement's correctness depends on other statements in the original question, the per-item labels are wrong.

What would settle it

Have a legal expert blindly re-answer a random sample of 100–200 decomposed statements using the official answer key and compare with the dataset's labels; systematic disagreement would invalidate the benchmark. A second check: run a strong model on the original multiple-choice questions and compare with its per-statement accuracy; a large gap would show that the decomposition changes the task's difficulty.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Future work on Japanese legal AI can use JBE-QA to compare models on a single shared test, per subject and per item.
The consistent difficulty ranking (Constitution easier than Civil/Penal Code) suggests that single-subject benchmarks, such as earlier Civil Code resources, overstate overall legal ability.
The case study indicates that models lack knowledge of precedent-based statutory interpretation, giving a concrete target for legal LLM training.
The result that reasoning-enabled proprietary models lead, while Japanese-specialised open models gain little, implies that inference-time reasoning is currently a stronger lever than language-specific adaptation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the dataset is drawn from public official exams, one natural next test is whether any model has memorized the answers; holding out recent years would reveal whether the reported gains hold on unseen items.
The decomposition removes the original option-combination structure: the exam asks examinees to pick the correct combination, which is a different and possibly harder task than judging statements one by one. A follow-up study could quantify how much harder the original format is for models.
The 52 excluded questions, mostly from the Penal Code, mean the Penal Code portion may not be fully representative of the original exam; subject comparisons should be read with that caveat.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful new Japanese legal benchmark; subject-difficulty claim needs per-subject label-balance checks.

read the letter

Quick take: this paper deserves a serious referee. The dataset is genuinely new and useful. The headline evaluation claims—especially 'Constitution is easier'—are not yet secure, but that's fixable.

What's actually new: JBE-QA is the first Japanese legal benchmark covering Civil Code, Penal Code, and Constitution from actual bar exams. Prior resources (COLIEE, Choi et al., JTD) were Civil Code only. The decomposition into binary true/false judgments with structured fields is a sensible way to turn the multiple-choice format into a clean evaluation task, and the construction is transparent: they describe the PDF-to-XML pipeline, say which questions were excluded and why, and did manual review. The baseline sweep across 26 models is useful, though it's a snapshot.

Credit where due: the dataset itself is the contribution, and it looks carefully built. The paper also honestly flags limitations: it only covers tanto-shiki (knowledge-focused) questions, not the essay portion, and labels may go stale when laws change. That's the right attitude.

Soft spots, in proportion: the stress-test concern is real. Table 3 compares F1 across subjects, but the paper only reports overall label balance (47.6% True). If the Constitution subset has a different True/False ratio than the Penal Code subset—and there's no reason the official exam would balance them within subject—then F1 differences can reflect label prevalence and subject-specific bias, not legal knowledge. The prompt explicitly names the subject, which makes a bias plausible. The claim that Constitution is easier should be backed by per-subject label distributions and a prevalence-controlled metric (AUROC or balanced accuracy) before I'd trust it.

Other soft spots: single-run evaluations with no confidence intervals; the four-shot exemplars are sampled but not released; and the exclusion of 52 questions—46 from Penal Code—could bias subject-level comparisons. These are all addressable and don't undermine the dataset's value.

For whom: legal NLP researchers who want a Japanese-language benchmark, and anyone doing multilingual legal evaluation. It won't change the field's direction, but it fills a real gap.

Peer review: yes, send it out. The dataset deserves referee time. I'd ask for per-subject statistics and a robustness check on the evaluation claims, but the core resource is solid.

Referee Report

2 major / 6 minor

Summary. This paper introduces JBE-QA, a Japanese legal-domain QA dataset derived from the multiple-choice section of the Japanese bar exam (2015–2024). Original questions are decomposed into individual true/false statements, yielding 3,464 items with balanced labels across the Civil Code, the Penal Code, and the Constitution, with optional instruction, theme, lead-in, and remark fields. The authors evaluate 26 proprietary and open-weight LLMs in zero- and four-shot settings, reporting F1 and faithfulness scores. They conclude that proprietary models with reasoning enabled perform best and that Constitution questions are generally easier than Civil Code or Penal Code questions.

Significance. If the dataset labels are accurate, JBE-QA fills a real gap: existing Japanese legal datasets focus on the Civil Code, so a multi-subject resource is a useful contribution for evaluating LLMs on Japanese legal knowledge. The decomposition into binary judgments and the public release make evaluation straightforward and reproducible, and the 26-model baseline is broad. The dataset construction is described in detail, with exclusions disclosed and a manual review step. However, the headline subject-difficulty and model-ranking claims are currently supported only by aggregate F1 scores from single runs; they need strengthening before the paper can be accepted as is.

major comments (2)

[§4.1, Table 3] The claim that Constitution questions are generally easier than Civil Code or Penal Code questions rests on comparing F1 scores across subjects. §2.5 reports label balance only for the whole dataset (47.6% True, 52.4% False), not per subject. F1 for the positive class is prevalence-sensitive: a subject with a higher True ratio yields a higher F1 even if per-class sensitivity/specificity are identical. Because the prompt states the subject (§3.2), models may apply subject-specific prior biases. To secure the subject-difficulty finding, report per-subject label distributions and a prevalence-invariant metric (balanced accuracy, AUC, or MCC), or show that the ordering survives when prevalence is controlled.
[§3.2, Table 2] All evaluations are single-run; reasoning models are sampled at temperature 1 or default, so their outputs are stochastic. Top-of-table differences are small (e.g., Opus 4.1 w/ Reasoning 0.814 vs GPT-5 0.794 vs Sonnet 4 w/ Reasoning 0.780 in zero-shot F1). Without multiple runs, confidence intervals, or significance tests, the headline ranking of reasoning models is not established. Please report repeated runs (at least 3–5) and variance, or use deterministic decoding where available, and temper claims accordingly.

minor comments (6)

[§3] The 'training set' is never defined; the dataset appears to have no train/test split. Clarify how the four-shot exemplars are sampled and whether they come from the full dataset.
[§3.1, Table 2] Model naming is inconsistent: 'GPT-4o-11' is used in Table 2 instead of the full checkpoint name, and §4.2 refers to 'Opus-4 (w/ Reasoning)' while Table 2 lists 'Opus 4.1 (w/ Reasoning)'.
[§2.5, Table 1] Table 1 would be more informative with True/False counts per subject in addition to the overall balance; this is directly relevant to the F1 comparison in §4.1.
[§3] The F1 metric is not fully defined. State explicitly that the positive class is 'True' and specify whether the reported F1 is the F1 of that class; the 'random guessing' value of 0.488 in §4.1 assumes this.
[§1, §6] 'First comprehensive benchmark' is somewhat overstated because only three of the JBE's multiple-choice subjects are included. Consider wording such as 'first benchmark covering these three core subjects' and discuss future expansion to the remaining subjects.
[§2.4] Quality assurance relies on manual review by the authors, but no inter-annotator agreement or label-error rate is reported. Reporting the number of corrected instances or a small independent audit would increase confidence in label reliability.

Circularity Check

0 steps flagged

No significant circularity: JBE-QA is an externally sourced benchmark with measured LLM performance, not a derivation from fitted parameters or self-referential constructions.

full rationale

The paper's central contributions are a dataset constructed from official Japanese bar exam materials and an empirical evaluation of 26 LLMs on that dataset. The gold labels are taken from the official exam answers, not derived from model predictions or from any fitted quantity; the decomposition into binary statements is a data-formatting step, and the paper explicitly reports manual review to correct extraction errors (§2.3–2.4). The evaluation results (model rankings, subject difficulty comparisons) are reported as measurements from the benchmark, not as predictions generated by a model whose parameters were fit to those same results. No self-citation is invoked as load-bearing evidence for the central empirical claims; citations to prior work (e.g., COLIEE, Choi et al.) are used only for context and comparison. The reported claim that Constitution questions are easier is an empirical observation on this dataset; potential concerns such as per-subject label-prevalence differences affecting F1 comparisons are validity or robustness issues, not circularity. The paper's own limitation statement about future law changes concerns label maintenance, not circular construction. Under the enumerated criteria, no step reduces to its own inputs by definition, by fitted-parameter renaming, or by self-referential author-authority chains.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No fitted constants appear; the claims are empirical measurements over a new dataset. The key load-bearing assumptions are the fidelity of the official labels, the validity of binary decomposition, and the stability of single-run model evaluations.

axioms (4)

domain assumption The official Ministry of Justice answers to the past bar exam are correct gold labels for every decomposed statement.
The dataset uses the official tantō-shiki answer key directly as ground truth (§2.3–2.4); no independent legal validation is reported.
domain assumption Each statement in the original multi-statement multiple-choice question has a well-defined truth value independent of the other statements, so the combination-based answer can be split into isolated binary items.
The conversion (§2.2) assumes question decomposition preserves labels; 52 questions that resist this format are dropped (§2.3).
ad hoc to paper Single-run evaluations at temperature 0 (non-reasoning) or temperature 1 (reasoning) are sufficient to compare LLM performance reliably.
No repeated runs, seeds, or confidence intervals are provided (§3.2); the comparative claims depend on this assumption.
ad hoc to paper The four randomly sampled exemplars in the four-shot setting are representative and do not systematically distort model outputs.
Exemplars are randomly drawn (§3.2) but not fixed or released; LLM-jp models show performance collapses attributed to exemplar content.

pith-pipeline@v1.3.0-alltime-deepseek · 13706 in / 12850 out tokens · 114211 ms · 2026-08-03T19:38:03.164112+00:00 · methodology

0 comments

read the original abstract

We introduce JBE-QA, a Japanese Bar Exam Question-Answering dataset to evaluate large language models' legal knowledge. Derived from the multiple-choice (tanto-shiki) section of the Japanese bar exam (2015-2024), JBE-QA provides the first comprehensive benchmark for Japanese legal-domain evaluation of LLMs. It covers the Civil Code, the Penal Code, and the Constitution, extending beyond the Civil Code focus of prior Japanese resources. Each question is decomposed into independent true/false judgments with structured contextual fields. The dataset contains 3,464 items with balanced labels. We evaluate 26 LLMs, including proprietary, open-weight, Japanese-specialised, and reasoning models. Our results show that proprietary models with reasoning enabled perform best, and the Constitution questions are generally easier than the Civil Code or the Penal Code questions.

Figures

Figures reproduced from arXiv: 2511.22869 by Fumihito Nishino, Hiroaki Yamada, Ken Satoh, Nguyen Ha Thanh, Yusuke Miyao, Zhihan Cao.

discussion (0)

Reference graph

Works this paper leans on

14 extracted references · 3 linked inside Pith

[1]

Introduction Legal practice relies heavily on the individual pro- fessional’s expertise and experience. This expert- dependency raises the need for legal information processing, which, for example, helps retrieve le- gal documents efficiently and analyse their argu- mentative structures (Nguyen et al., 2024; Vuong et al., 2023). In this way, legal informa...

Pith/arXiv arXiv 2024
[2]

Ignore damages, and enforcement costs

Dataset Construction Our dataset is derived from the Japanese Bar Ex- amination (JBE)1, specifically from the multiple- choice type questions,tant¯o-shikitest. All the data are in the Japanese language. 2.1. Japanese Bar Exam JBE is a national examination that grants the legal qualifications required to become a judge, prose- cutor, or attorney in Japan. ...

2024
[3]

extended thinking

Baseline Experiment Each instance in the JBE-QA dataset represents a binary classification problem. We instruct an LLM under evaluation to only output a binary truth value as the prediction for every instance (c.f. Sec- tion 3.2). However, some models may fail to fol- low the instruction, resulting in non-binary outputs. In such cases, the prediction is s...

2023
[4]

1. Explanation: According to the position of the case law, this case consti- tutes the crime of evidence destruction

Baseline Results Table 2 presents the F1 and Faithfulness scores on the whole dataset, under zero- and four-shot settings, grouped by whether a model is open (O), is Japanese-specialised (J), and supports reason- ing (R). 8The formatter script is available online. 9They are usually 1. Some models, for example GPT- 5 and o3, do not support adjustment of te...
[5]

ForEnglishresources,LexGLUE(Chalkidisetal., 2022), provided a suite of tasks for legal lan- guage understanding

Related Work There are different benchmarks, from focusing on single tasks in specific jurisdictions to employing multiple tasks, languages, and legal systems, re- flecting the growing needs of computational solu- tions for the legal domain. ForEnglishresources,LexGLUE(Chalkidisetal., 2022), provided a suite of tasks for legal lan- guage understanding. Th...

2022
[6]

Conclusion We introduce JBE-QA, a dataset for evaluating LLMs’ legal knowledge. It covers the Civil Code, thePenalCode,andtheConstitution,extendingbe- yondtheCivilCodefocusofthepreviousJapanese legal resources, and hence enabling a compre- hensive legal knowledge evaluation. We bench- mark26models,includingproprietary,open-weight, Japanese-specialised, an...
[7]

Our source of the dataset is an official archive of the past Japanese bar exam, and it does not contain any private infor- mation or confidential information

Ethics Statement We believe that this dataset does not contain any significant ethical concerns. Our source of the dataset is an official archive of the past Japanese bar exam, and it does not contain any private infor- mation or confidential information. Currently, there are no feasible use cases for this dataset that can involve potential harm or ethical issues
[8]

Thetant ¯o-shikiquestions require relatively fewer reasoning steps in order to apply the knowledge to solve the questions, com- pared withronbun-shikiquestions

Limitations Our dataset primarily focuses on the fundamental legal knowledge of the Japanese legal system us- ingtant ¯o-shikiquestions. Thetant ¯o-shikiquestions require relatively fewer reasoning steps in order to apply the knowledge to solve the questions, com- pared withronbun-shikiquestions. In the practical situation,moreintensivelegalreasoningisreq...
[9]

R&D Hub Aimed at Ensuring Transparency and Reliability of Gen- erative AI Models

Acknowledgement This work was supported by the “R&D Hub Aimed at Ensuring Transparency and Reliability of Gen- erative AI Models” project of the Ministry of Edu- cation, Culture, Sports, Science and Technology, and by JST PRESTO, Japan, Grant Number JP- MJPR236B
[10]

Bibliographical References ABEJA. 2025. Abeja-qwen2.5-32b-japanese-v0.1: A japanese language model based on qwen2.5- 32b-instruct. Pawitsapak Akarajaradwong, Pirat Pothavorn, Chompakorn Chaksangchaichot, Panuthep Ta- sawong, Thitiwat Nopparatbundit, Keerakiat Pratai, and Sarana Nutanong. 2025. Nitibench: A comprehensive study of llm framework capa- biliti...

Pith/arXiv arXiv 2025
[12]

InProceedings of the 31st International Conference on Computational Lin- guistics, pages 11135–11160

Nyayaanumana and inlegalllama: The largest indian legal judgment prediction dataset and specialized language model for enhanced decision analysis. InProceedings of the 31st International Conference on Computational Lin- guistics, pages 11135–11160. Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Sturmer, and Ilias Chalkidis
[14]

Hiroaki Yamada, Takenobu Tokunaga, Ryutaro Ohara, AkiraTokutsu, KeisukeTakeshita, andMi- hoko Sumida

Sm-bert-cr: a deep learning approach for caselawretrievalwithsupportingmodel.Artificial Intelligence and Law, 31(3):601–628. Hiroaki Yamada, Takenobu Tokunaga, Ryutaro Ohara, AkiraTokutsu, KeisukeTakeshita, andMi- hoko Sumida. 2024. Japanese tort-case dataset forrationale-supportedlegaljudgmentprediction. Artificial Intelligence and Law, pages 1–25. Evi Y...

2024
[2023]

Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Naka- mura, Mengsay Loem, Rio Yokota, and Sakae Mizuki

Lextreme: A multi-lingual and multi-task benchmark for the legal domain. Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Naka- mura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a large japanese web cor- pus for large language models. InProceedings of the First Conference on Language Modeling, COL...

Pith/arXiv arXiv 2024
[2025]

In Korea, LBOX OPEN (Hwang et al., 2022) and KBL (Kimyeeun et al., 2024) are prominent multi-task benchmarks

is a large judgment prediction dataset. In Korea, LBOX OPEN (Hwang et al., 2022) and KBL (Kimyeeun et al., 2024) are prominent multi-task benchmarks. Southeast Asian lan- guages are also represented, with benchmarks for Thai (NitiBench, Akarajaradwong et al., 2025), Viet- namese(VLQA,Nguyenetal.,2025b), andIndone- sian (IndoLER, Yulianti et al., 2024). In...

2022

[1] [1]

Introduction Legal practice relies heavily on the individual pro- fessional’s expertise and experience. This expert- dependency raises the need for legal information processing, which, for example, helps retrieve le- gal documents efficiently and analyse their argu- mentative structures (Nguyen et al., 2024; Vuong et al., 2023). In this way, legal informa...

Pith/arXiv arXiv 2024

[2] [2]

Ignore damages, and enforcement costs

Dataset Construction Our dataset is derived from the Japanese Bar Ex- amination (JBE)1, specifically from the multiple- choice type questions,tant¯o-shikitest. All the data are in the Japanese language. 2.1. Japanese Bar Exam JBE is a national examination that grants the legal qualifications required to become a judge, prose- cutor, or attorney in Japan. ...

2024

[3] [3]

extended thinking

Baseline Experiment Each instance in the JBE-QA dataset represents a binary classification problem. We instruct an LLM under evaluation to only output a binary truth value as the prediction for every instance (c.f. Sec- tion 3.2). However, some models may fail to fol- low the instruction, resulting in non-binary outputs. In such cases, the prediction is s...

2023

[4] [4]

1. Explanation: According to the position of the case law, this case consti- tutes the crime of evidence destruction

Baseline Results Table 2 presents the F1 and Faithfulness scores on the whole dataset, under zero- and four-shot settings, grouped by whether a model is open (O), is Japanese-specialised (J), and supports reason- ing (R). 8The formatter script is available online. 9They are usually 1. Some models, for example GPT- 5 and o3, do not support adjustment of te...

[5] [5]

ForEnglishresources,LexGLUE(Chalkidisetal., 2022), provided a suite of tasks for legal lan- guage understanding

Related Work There are different benchmarks, from focusing on single tasks in specific jurisdictions to employing multiple tasks, languages, and legal systems, re- flecting the growing needs of computational solu- tions for the legal domain. ForEnglishresources,LexGLUE(Chalkidisetal., 2022), provided a suite of tasks for legal lan- guage understanding. Th...

2022

[6] [6]

Conclusion We introduce JBE-QA, a dataset for evaluating LLMs’ legal knowledge. It covers the Civil Code, thePenalCode,andtheConstitution,extendingbe- yondtheCivilCodefocusofthepreviousJapanese legal resources, and hence enabling a compre- hensive legal knowledge evaluation. We bench- mark26models,includingproprietary,open-weight, Japanese-specialised, an...

[7] [7]

Our source of the dataset is an official archive of the past Japanese bar exam, and it does not contain any private infor- mation or confidential information

Ethics Statement We believe that this dataset does not contain any significant ethical concerns. Our source of the dataset is an official archive of the past Japanese bar exam, and it does not contain any private infor- mation or confidential information. Currently, there are no feasible use cases for this dataset that can involve potential harm or ethical issues

[8] [8]

Thetant ¯o-shikiquestions require relatively fewer reasoning steps in order to apply the knowledge to solve the questions, com- pared withronbun-shikiquestions

Limitations Our dataset primarily focuses on the fundamental legal knowledge of the Japanese legal system us- ingtant ¯o-shikiquestions. Thetant ¯o-shikiquestions require relatively fewer reasoning steps in order to apply the knowledge to solve the questions, com- pared withronbun-shikiquestions. In the practical situation,moreintensivelegalreasoningisreq...

[9] [9]

R&D Hub Aimed at Ensuring Transparency and Reliability of Gen- erative AI Models

Acknowledgement This work was supported by the “R&D Hub Aimed at Ensuring Transparency and Reliability of Gen- erative AI Models” project of the Ministry of Edu- cation, Culture, Sports, Science and Technology, and by JST PRESTO, Japan, Grant Number JP- MJPR236B

[10] [10]

Bibliographical References ABEJA. 2025. Abeja-qwen2.5-32b-japanese-v0.1: A japanese language model based on qwen2.5- 32b-instruct. Pawitsapak Akarajaradwong, Pirat Pothavorn, Chompakorn Chaksangchaichot, Panuthep Ta- sawong, Thitiwat Nopparatbundit, Keerakiat Pratai, and Sarana Nutanong. 2025. Nitibench: A comprehensive study of llm framework capa- biliti...

Pith/arXiv arXiv 2025

[11] [12]

InProceedings of the 31st International Conference on Computational Lin- guistics, pages 11135–11160

Nyayaanumana and inlegalllama: The largest indian legal judgment prediction dataset and specialized language model for enhanced decision analysis. InProceedings of the 31st International Conference on Computational Lin- guistics, pages 11135–11160. Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Sturmer, and Ilias Chalkidis

[12] [14]

Hiroaki Yamada, Takenobu Tokunaga, Ryutaro Ohara, AkiraTokutsu, KeisukeTakeshita, andMi- hoko Sumida

Sm-bert-cr: a deep learning approach for caselawretrievalwithsupportingmodel.Artificial Intelligence and Law, 31(3):601–628. Hiroaki Yamada, Takenobu Tokunaga, Ryutaro Ohara, AkiraTokutsu, KeisukeTakeshita, andMi- hoko Sumida. 2024. Japanese tort-case dataset forrationale-supportedlegaljudgmentprediction. Artificial Intelligence and Law, pages 1–25. Evi Y...

2024

[13] [2023]

Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Naka- mura, Mengsay Loem, Rio Yokota, and Sakae Mizuki

Lextreme: A multi-lingual and multi-task benchmark for the legal domain. Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Naka- mura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a large japanese web cor- pus for large language models. InProceedings of the First Conference on Language Modeling, COL...

Pith/arXiv arXiv 2024

[14] [2025]

In Korea, LBOX OPEN (Hwang et al., 2022) and KBL (Kimyeeun et al., 2024) are prominent multi-task benchmarks

is a large judgment prediction dataset. In Korea, LBOX OPEN (Hwang et al., 2022) and KBL (Kimyeeun et al., 2024) are prominent multi-task benchmarks. Southeast Asian lan- guages are also represented, with benchmarks for Thai (NitiBench, Akarajaradwong et al., 2025), Viet- namese(VLQA,Nguyenetal.,2025b), andIndone- sian (IndoLER, Yulianti et al., 2024). In...

2022