When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoning

Claire Barale; Michael Rovatsos; Nehal Bhuta

REVIEW 5 major objections 5 minor 1 cited by

When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoning

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Standard ML fairness audits cannot distinguish bias from legally justified variation in refugee rulings.

desk verdict Useful divergence finding, but the claim that ML cannot evaluate legal fairness overshoots the evidence. read the letter →

arxiv 2506.03913 v1 pith:DIGLZUVK submitted 2025-06-04 cs.CL cs.LG

Claire Barale , Michael Rovatsos , Nehal Bhuta This is my paper · ORCID

classification cs.CLcs.LG

keywords fairnessevaluationrefugeeadjudicationlegalreasoningmachinelearningsemanticclusteringpredictivemodelingfeature-basedanalysisprocedural

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that standard machine-learning methods cannot evaluate fairness in discretionary legal domains such as refugee adjudication, because they capture statistical patterns without capturing legal reasoning or justification. Testing three common approaches on 59,000+ full-text Canadian refugee decisions (AsyLex), the authors find that feature-based statistical analysis, semantic clustering, and predictive modeling yield divergent and sometimes contradictory signals. Predictive models reach high accuracy but lean on procedural cues such as hearing date and judge identity rather than legally salient reasons; semantic clusters track surface language and outcomes, not legal grounds. The paper concludes that statistical regularity is not fairness in law, and that fairness evaluation must incorporate legal reasoning and institutional context rather than outcome parity alone. A sympathetic reader would care because these methods are increasingly used to audit high-stakes legal systems, and the paper argues they can mislabel justifiable discretion as bias.

What carries the argument

The machinery is the side-by-side comparison of three method families on the same corpus, framed by the legal distinction between distributive fairness (similar outcomes for similar cases) and procedural or justificatory fairness (whether decisions are reasoned, consistent in process, and normatively defensible). Method 1 runs hypothesis-driven statistical tests over hand-extracted categorical features; Method 2 embeds full decision texts with the text-embedding-3-small model and clusters them with K-means at k=2; Method 3 trains a random forest on structured feature subsets and a two-layer feedforward neural network on free text, evaluating with macro F1, ROC AUC, accuracy, and the Equality of Opportunity criterion. The conceptual load-bearing idea is that disparity becomes unfairness only when outcome differences cannot be accounted for by legitimate, legally salient factors, so a method that only measures outcome variation cannot by itself deliver a fairness verdict. The divergence across methods is itself the finding: it shows the absence of a reliable evaluative signal and each method's implicit, often false, assumption about what similarity or predictability means.

What would settle it

Falsify the claim by building a method that demonstrably captures legal reasoning: for example, a model trained only on the final reasoning paragraphs of AsyLex decisions whose clusters align with legal grounds and justifications rather than outcome or judge identity, or whose feature importances are dominated by legal grounds rather than hearing date; more directly, any ML system that, on an expert-annotated set of decisions with known justifiability, reliably separates justified from unjustified disparities would refute the paper's blanket negative conclusion.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is negative but pointed: none of the three canonical ML approaches can tell unjust bias from legally justified variation in refugee status determinations. Feature-based analysis detects stark disparities across judges, cities, years, and claimant groups but cannot explain their causes or say whether they are legitimate. Semantic clustering splits decisions along outcome and judge-language lines yet shows no significant structure for gender, minors, LGBTQIA+ status, time, or city, and only partial, non-robust alignment with legal grounds. Predictive modeling achieves up to 93.8% accuracy, but feature importance is dominated by the hearing date, judge identity, and other procedural variables, while legal justifications like credibility and persecution ground rank low; the Equality of Opportunity metric shows group recall gaps that cannot be interpreted without ground truth on justifiability. Because the three methods disagree on key dimensions, the authors argue that no consistent, reliable evaluative signal exists, and that current computational fairness assessment in law is therefore unsound.

Load-bearing premise

The load-bearing premise is that the three specific implementations — a generic text embedding with a fixed two-cluster split, a random forest over hand-defined features, and a minimal two-layer neural network — adequately represent machine-learning fairness evaluation, so that no better-tuned or legally informed variant of any of them could capture the legal reasoning these three miss.

Editorial extensions

If this is right

If the paper is right, existing ML-based fairness audits of refugee adjudication and similar discretionary legal systems cannot support conclusions of bias or unfairness; flagging a judge or city for disparate outcomes would require a causal and justificatory analysis these methods do not provide.
High predictive accuracy for legal outcomes should not be read as evidence that a system is fair or that decisions are regular in a normatively good sense, since accuracy is driven by procedural artifacts such as hearing date and judge identity.
Fairness metrics like Equality of Opportunity, when applied to legal decisions without ground truth on the correctness or justifiability of outcomes, produce uninterpretable group gaps that may reflect case composition rather than discrimination.
A defensible computational fairness evaluation in law would need to measure the structure and adequacy of legal justification, for example via text-level annotations of reasons, rather than outcome parity alone.
Because the three methods disagree on whether temporal, geographic, and demographic disparities exist, any single-method audit is unreliable; the choice of method can determine the conclusion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's negative conclusion suggests a positive research program: train models to evaluate the adequacy of justification rather than the outcome, for example by predicting whether a decision's stated reasons support its conclusion using expert annotations of justifiability as ground truth, and the AsyLex corpus could be extended for this.
The dominance of hearing date as a predictor hints at a confound between case-mix shifts over time and judicial behavior; a natural experiment exploiting quasi-random judge assignment could separate these causes, something the correlational methods in the paper cannot do.
The clustering method's failure to find demographic or temporal structure may reflect the generic embedding rather than the absence of signal; topic modeling or legal-domain fine-tuned embeddings over the reasoning paragraphs of decisions is a testable extension that could either reinforce or bound the paper's claim.
The paper's distinction between justified and unjustified disparity could be formalized as a counterfactual question — would the same case with a different judge, city, or year have received the same outcome? — linking legal fairness evaluation to causal inference from text, a connection the paper notes but does not develop.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Let me give you the short version: this paper is worth reading but not for the headline conclusion. It shows something real — three standard ML fairness-evaluation methods on 59k refugee decisions give divergent, often contradictory signals. That is a useful data point for anyone building or auditing fairness tools in law. But the paper's stronger claim, that ML methods are fundamentally limited and cannot evaluate legal fairness, goes beyond what the experiments can support.

What's actually new: it is the first head-to-head comparison of feature-based statistical analysis, semantic clustering, and predictive modeling on full-text refugee decisions. The AsyLex corpus is a substantial real-world resource. I also credit the authors for repeatedly acknowledging the key limitations — no ground truth, clustering sensitivity to embeddings and hyperparameters, the possibility that disparities reflect justified discretion. They cite the refugee-roulette literature and don't oversell parity metrics.

The soft spots are where the stress-test note lands. The controlled subset is selected by taking judges with extreme grant rates and then 'validating' the split with a chi-square test on the same outcome data. The p=1.76e-27 is just confirming the selection rule, not an independent check. The clustering uses a fixed k=2 with no validation; on an imbalanced outcome that split is likely to separate by outcome regardless of legal content. The random forest's Gini importance is biased under correlated features, and the authors know it. And the neural network that hits 99.8% AUC on full text is never interpreted, so claiming it ignores legal reasoning is unsupported. Because there are no ground-truth labels of justifiability, the paper observes correlations and then asserts that the intended construct is missed. That is the main logical gap.

None of this kills the paper. The empirical divergence is there, and the call for justification-aware evaluation methods is a reasonable direction. The conclusion just needs to be recalibrated from 'ML cannot evaluate legal fairness' to 'these three off-the-shelf implementations cannot, and here is where they fail.' I would send this to a serious referee — it deserves careful review, and a good referee can push on exactly these points. The paper will be cited for the divergence result, and the discussion is a fair starting point for the field.

For a reading group: I would bring it to discuss the method choices and the gap between statistical signal and legal meaning. There's enough substance to argue over.

Referee Report

5 major / 5 minor

Summary. The paper argues that standard machine learning techniques cannot reliably evaluate fairness in legally discretionary domains such as refugee adjudication, because they fail to capture substantive legal reasoning and justification. Using the AsyLex dataset of over 59,000 Canadian refugee decisions, the authors apply three methods—feature-based statistical analysis, semantic clustering with embeddings and k-means, and predictive modeling with random forests and a neural network—and report that the methods produce divergent and sometimes contradictory signals. They conclude that none of the methods can distinguish unjust bias from legally justified variation, and advocate for fairness evaluation methods grounded in legal reasoning and institutional context. The paper includes a controlled subset of six judges with extreme grant rates, feature importance analyses, group-level fairness metrics, and a synthesis table comparing the three methods.

Significance. If the central claim were fully supported, the paper would be a valuable corrective to naive applications of ML fairness metrics in high-stakes legal settings, and the AsyLex dataset is a useful public resource for studying refugee adjudication. The paper's emphasis on the gap between statistical disparity and legal unfairness, and its explicit attention to limitations of each method, are strengths. However, the empirical evidence as presented does not establish the strong negative conclusion: the controlled subset is constructed to confirm itself, the clustering analysis fixes k=2 without validation, the feature importance evidence rests on a biased measure, and no ground truth for 'legally justified' decisions is used to test the target construct. The paper is best read as a case study of three specific configurations, not as proof that ML cannot evaluate legal fairness.

major comments (5)

[Dataset (Controlled Subset)] The chi-square test (p = 1.76 × 10^-27) is used to validate the construction of the LGR and HGR judge groups, but those groups were defined by their grant rates: LGR judges issued exclusively negative decisions and HGR judges had above-average grant rates. The test therefore restates the selection criterion and provides no independent confirmation. Claims built on this subset—such as the clustering separation in Method 2 and the improved predictive performance in Table 1—should be interpreted as consequences of the selection rule, not as discoveries about legal fairness.
[Method 2 (Semantic Clustering)] The number of clusters is fixed at k=2 with no validation (no silhouette, gap statistic, or stability analysis). The paper itself acknowledges that semantic clustering is sensitive to embedding models, clustering algorithms, and hyperparameters, but no alternatives are tested. The conclusion that semantic clustering 'fails to capture substantive legal reasoning' is therefore demonstrated only for one configuration; a different embedding or a validated cluster count could plausibly align clusters with legal grounds or justifications. Either cluster validation should be reported or the claim should be restricted to the specific configuration used.
[Method 3 (Predictive Modeling), Fig. 4] The claim that predictive models depend on contextual and procedural features rather than legally salient features relies on Gini impurity importance from the random forest. Gini importance is known to be biased when features are correlated, and the paper acknowledges 'the potential non-independence of the features used' in the limitations of Method 3, yet still treats Fig. 4 as load-bearing evidence. Additionally, the neural network achieves near-perfect discrimination (99.8% AUC for women in Table 2), but its internal representations are never analyzed, so there is no direct evidence for the assertion that it ignores legal reasoning. Permutation importance, SHAP values, or analysis of the neural network's learned representations would be needed to support these claims.
[Dataset and Problem Setup / Discussion] The central claim is that ML methods 'cannot reliably evaluate legal fairness' because they fail to capture legal reasoning and justification. However, the paper explicitly states that the methods are compared to one another rather than to a ground truth, and no labels for 'legally justified' versus 'unjustified' decisions are used in the evaluation. Without an operationalization or gold standard for the target construct, the experiments can show that the methods produce divergent signals, but they cannot establish that the methods miss legal reasoning. The conclusion should be reframed as evidence that these particular implementations do not yield consistent, interpretable fairness signals, not that ML in general cannot evaluate legal fairness.
[Methodology and Experimental Setup] The three implementations are presented as representative of 'common ML approaches' for fairness evaluation, but no evidence is offered that the specific choices—OpenAI text-embedding-3-small, k-means with k=2, a random forest with Gini importance, and a two-layer neural network—are a representative sample of the space of ML fairness evaluation methods. The abstract's strong conclusion ('current computational approaches fall short') extrapolates from these three configurations. The paper should either test a broader set of models, embeddings, cluster counts, and feature sets, or explicitly restrict the negative conclusion to the configurations studied.

minor comments (5)

[Contributions] The first contribution contains a typo: 'a dataset that has no been studied' should read 'has not been studied.'
[Throughout] The dataset name is inconsistently formatted as 'ASYLEX' and 'A SYLEX', and 'CLUSTER' appears as 'C LUSTER' in several places. Please use consistent typography (e.g., 'AsyLex' and 'Cluster').
[Table 1] The feature subset labels are inconsistent between the text and the table: the text lists Part 3 as 'Explanations' and Part 6 as 'Locations', but the table rows list Part 3 as 'Locations' and Part 6 as 'Explanations'. Please align the numbering.
[Fig. 3] The caption uses 'Focused Subset' while the text uses 'Controlled Subset'; please use one term consistently.
[Methodology, Evaluation Metrics] The choice of Equality of Opportunity is said to be 'aligned with legal principles', but no legal source or detailed argument is provided to support this alignment; please justify or qualify this claim.

Circularity Check

1 steps flagged · score 2.0 of 10

One minor self-definitional validation of the Controlled Subset; the central negative claim about ML fairness evaluation is empirical and not circular.

self definitional [Dataset and Problem Setup, 'Controlled Subset' section (p. 4)]
"These judges were selected based on the number of decisions and grouped by decision outcomes: Judges 146, 465, and 497 (LGR) issued exclusively negative decisions, while Judges 280, 154, and 346 (HGR) had above-average grant rates. ... To validate our choice of judges, we conducted a two-sample significance test (chi-square). The low resulting p-value (1.76 × 10−27) confirms that the difference in grant rates between groups is statistically significant."

The LGR and HGR groups are constructed from the outcome variable: LGR judges issued exclusively negative decisions and HGR judges had above-average grant rates. The chi-square test then compares grant rates between these outcome-selected groups, so a significant difference is entailed by the selection rule. Reporting this as a 'validation' presents a property of the construction as independent empirical confirmation. The circularity is minor because this particular test is not used as evidence for the paper's central conclusion about ML methods' inability to capture legal reasoning.

full rationale

The paper's central claim—that feature-based analysis, semantic clustering, and predictive modeling do not reliably evaluate legal fairness because they fail to capture legal reasoning—is an empirical conclusion drawn from comparing three implemented methods on the AsyLex corpus. No fitted parameter is renamed as a fairness prediction, and the methods are not defined in terms of the conclusion. The only identifiable circular step is the 'validation' of the Controlled Subset: judges were selected into LGR/HGR groups by their grant rates, and the chi-square test then 'confirms' that the two groups differ in grant rates. That p-value is a mathematical consequence of the selection rule, not independent evidence. This step is not load-bearing for the paper's main argument, which does not depend on the chi-square validation. The paper's reliance on its own AsyLex annotations is a self-citation but is an external dataset resource, not an unverified theorem, so it does not constitute circularity under the stated rules. Overall, the derivation is self-contained apart from the minor self-definitional validation, giving a score of 2.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The paper's central claim rests on normative definitions of legal fairness, the representativeness of the AsyLex sample, the accuracy of the authors' prior annotations, and the appropriateness of specific modeling choices (k=2, EO, embedding model). No new physical or formal entities are introduced.

free parameters (3)

K-means cluster count (k) = 2
Chosen manually; no model selection criterion. The two-cluster split forces a binary partition that likely drives the cluster-outcome correlation.
Random forest hyperparameters = not reported
Tuned via stratified 5-fold cross-validation, but exact values (tree count, depth, etc.) are omitted, limiting reproducibility and making it unclear whether the feature-importance ranking is robust.
Semantic embedding model = OpenAI text-embedding-3-small
A commercial API model whose version and behavior are not pinned; clustering results are sensitive to this choice (acknowledged in Method 2 limitations).

assumptions (6)

domain assumption Legal fairness cannot be reduced to equal outcomes; it includes procedural and justificatory elements.
Stated in Introduction and Background; this normative premise defines what the paper means by fairness and underlies the claim that statistical methods are insufficient.
domain assumption Refugee law permits and sometimes requires outcome variation based on case-specific facts and credibility assessments.
Background section; used to argue that statistical disparity may be justified, so disparity detection does not equal unfairness.
domain assumption The AsyLex sample of publicly released decisions is adequate for comparing methods, even if not representative of all decisions.
Dataset section; the authors note sampling bias but assert it does not diminish their method-comparison conclusion.
domain assumption The manually annotated features from Barale et al. (2023) are accurate and legally relevant.
Feature-based analysis relies on these annotations; no re-annotation or inter-annotator agreement is reported for the subset used here.
ad hoc to paper Equality of Opportunity is an appropriate fairness metric for this domain.
Method 3 uses EO without justification that it captures legal fairness; the paper later argues such metrics are inadequate, creating tension.
ad hoc to paper K-means with k=2 yields a meaningful semantic structure.
Method 2; the paper selects two clusters and interprets them, but does not test other k values or cluster validity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoning." pith.science (2026). https://pith.science/paper/DIGLZUVK

@misc{pith2026250603913,
  author       = {Pith},
  title        = {Pith review of: When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DIGLZUVK}},
  note         = {Machine review of arXiv:2506.03913}
}

read the original abstract

Legal decisions are increasingly evaluated for fairness, consistency, and bias using machine learning (ML) techniques. In high-stakes domains like refugee adjudication, such methods are often applied to detect disparities in outcomes. Yet it remains unclear whether statistical methods can meaningfully assess fairness in legal contexts shaped by discretion, normative complexity, and limited ground truth. In this paper, we empirically evaluate three common ML approaches (feature-based analysis, semantic clustering, and predictive modeling) on a large, real-world dataset of 59,000+ Canadian refugee decisions (AsyLex). Our experiments show that these methods produce divergent and sometimes contradictory signals, that predictive modeling often depends on contextual and procedural features rather than legal features, and that semantic clustering fails to capture substantive legal reasoning. We show limitations of statistical fairness evaluation, challenge the assumption that statistical regularity equates to fairness, and argue that current computational approaches fall short of evaluating fairness in legally discretionary domains. We argue that evaluating fairness in law requires methods grounded not only in data, but in legal reasoning and institutional context.

Figures

Figures reproduced from arXiv: 2506.03913 by the authors.

**Figure 3.** Visualization of clusters and outcomes using t [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Aggregated feature importance for the full model [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

The Judge Variable: Challenging Judge-Agnostic Legal Judgment Prediction
cs.CL 2025-07 reject novelty 4.0 of 10

Models trained on individual judges' past child-custody rulings predict those judges' future rulings better than a model trained on all judges together, a result the paper reads as support for legal realism.

Reference graph

Works this paper leans on

27 extracted references · 19 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Barale, C.; Klaisoongnoen, M.; Minervini, P.; Rovatsos, M.; and Bhuta, N. 2023. A sy L ex: A Dataset for Legal Language Processing of Refugee Claims. In Proceedings of the Natural Legal Language Processing Workshop 2023, 244--257. Singapore: Association for Computational Linguistics

work page 2023
[4]

L.; and Eagel, J

Chen, D. L.; and Eagel, J. 2017. Can machine learning help predict the outcome of asylum adjudications? In Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law, 237--240

work page 2017
[5]

L.; and Loecher, M

Chen, D. L.; and Loecher, M. 2019. Mood and the malleability of moral reasoning. Available at SSRN 2740485

work page 2019
[6]

Dunn, M.; Sagun, L.; S irin, H.; and Chen, D. 2017. Early predictability of asylum court decisions. In Proceedings of the 16th Edition of the International Conference on Articial Intelligence and Law, ICAIL '17, 233–236. New York, NY, USA: Association for Computing Machinery. ISBN 9781450348911

work page 2017
[7]

Dwork, C.; Hardt, M.; Pitassi, T.; Reingold, O.; and Zemel, R. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, 214--226

2012
[8]

F.; Ho, D

Engstrom, D. F.; Ho, D. E.; Sharkey, C. M.; and Cu \'e llar, M.-F. 2020. Government by algorithm: Artificial intelligence in federal administrative agencies. NYU School of Law, Public Law Research Paper, (20-54)

work page 2020

Show all 27 references

[9]

Garg, A.; and Ma, M. 2025. Opportunities and Challenges in Legal AI. White paper, Stanford Law School. CodeX

2025
[10]

B.; Gummadi, K

Grgi \'c -Hla c a, N.; Zafar, M. B.; Gummadi, K. P.; and Weller, A. 2018. Beyond distributive fairness in algorithmic decision making: Feature selection for procedurally fair learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32

2018
[11]

Hardt, M.; Price, E.; and Srebro, N. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29

2016
[12]

Keith, K.; and Stent, A. 2019. Modeling Financial Analysts' Decision Making via the Pragmatics and Semantics of Earnings Calls. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 493--503. Florence, Italy: Association for Computational ...

2019
[13]

Legomsky, S. H. 2007. Learning to live with unequal justice: Asylum and the limits to consistency. Stan. L. Rev., 60: 413

2007
[14]

Mehrabi, N.; Morstatter, F.; Saxena, N.; Lerman, K.; and Galstyan, A. 2021. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6): 1--35

2021
[15]

Morse, L.; Teodorescu, M. H. M.; Awwad, Y.; and Kane, G. C. 2021. Do the ends justify the means? Variation in the distributive and procedural fairness of machine learning algorithms. Journal of Business Ethics, 1--13

2021
[16]

Passi, S.; and Barocas, S. 2019. Problem formulation and fairness. In Proceedings of the conference on fairness, accountability, and transparency, 39--48

2019
[17]

A.; Katsikouli, P.; Gammeltoft-Hansen, T.; and Slaats, T

Piccolo, S. A.; Katsikouli, P.; Gammeltoft-Hansen, T.; and Slaats, T. 2023. On predicting and explaining asylum adjudication. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, 217--226

2023
[18]

Preston, J. 2007. Big Disparities in Judging of Asylum Cases. The New York Times. Accessed: 2025-01-13

2007
[19]

I.; and Schrag, P

Ramji-Nogales, J.; Schoenholtz, A. I.; and Schrag, P. G. 2007. Refugee roulette: Disparities in asylum adjudication. Stan. L. Rev., 60: 295

2007
[20]

Rehaag, S. 2012. Judicial review of refugee determinations: The luck of the draw. Queen's LJ, 38: 1

2012
[21]

Rehaag, S. 2019. Judicial review of refugee determinations (II): Revisiting the luck of the draw. Queen's LJ, 45: 1

2019
[22]

Rehaag, S. 2023. Luck of the Draw III: Using AI to Examine Decision-Making in Federal Court Stays of Removal

2023
[23]

Rogers, A. 2021. Changing the World by Changing the Data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2182--2194. Online: Associat...

2021
[24]

Van der Maaten, L.; and Hinton, G. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(11)

2008
[25]

Wang, Y.; Xiao, C.; Ma, S.; Zhong, H.; Tu, C.; Zhang, T.; Liu, Z.; and Sun, M. 2021. Equality before the law: Legal judgment consistency analysis for fairness. arXiv preprint arXiv:2103.13868

2021 arXiv
[26]

Wang, Z.; Huang, C.; and Yao, X. 2024. Procedural fairness in machine learning. arXiv preprint arXiv:2404.01877

2024
[27]

Zhao, Y.; Wang, Y.; and Derr, T. 2023. Fairness and explainability: Bridging the gap towards fair model explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 11363--11371

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Barale, C.; Klaisoongnoen, M.; Minervini, P.; Rovatsos, M.; and Bhuta, N. 2023. A sy L ex: A Dataset for Legal Language Processing of Refugee Claims. In Proceedings of the Natural Legal Language Processing Workshop 2023, 244--257. Singapore: Association for Computational Linguistics

work page 2023

[4] [4]

L.; and Eagel, J

Chen, D. L.; and Eagel, J. 2017. Can machine learning help predict the outcome of asylum adjudications? In Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law, 237--240

work page 2017

[5] [5]

L.; and Loecher, M

Chen, D. L.; and Loecher, M. 2019. Mood and the malleability of moral reasoning. Available at SSRN 2740485

work page 2019

[6] [6]

Dunn, M.; Sagun, L.; S irin, H.; and Chen, D. 2017. Early predictability of asylum court decisions. In Proceedings of the 16th Edition of the International Conference on Articial Intelligence and Law, ICAIL '17, 233–236. New York, NY, USA: Association for Computing Machinery. ISBN 9781450348911

work page 2017

[7] [7]

Dwork, C.; Hardt, M.; Pitassi, T.; Reingold, O.; and Zemel, R. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, 214--226

2012

[8] [8]

F.; Ho, D

Engstrom, D. F.; Ho, D. E.; Sharkey, C. M.; and Cu \'e llar, M.-F. 2020. Government by algorithm: Artificial intelligence in federal administrative agencies. NYU School of Law, Public Law Research Paper, (20-54)

work page 2020