{"id":"9e228c76-ec45-47bd-9171-e4a61db69c73","arxiv_id":"2608.10400","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Most Harris County bail magistrates can be modeled by small decision trees, but each judge uses different rules, producing near-random between-judge consistency.","lead":"The paper uses interpretable machine learning to test whether magistrate judges in Harris County, Texas decide bail cases by following simple rule-like formulas, and finds that most judges can be approximated by small decision trees. However, the rules differ sharply across judges, so a defendant's likelihood of release depends heavily on which judge hears the case.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'judges behave algorithmically' claim is an artifact of removing the 18% of cases where judges used case-specific information; full-data results are needed before the headline is justified.","rationale":"The strongest claim is the empirical headline: 'judges generally behave algorithmically' because small decision trees capture at least 85% of decisions for 17 of 21 judges. That claim is computed after removing unexplainable cases, and unexplainable cases include exactly the individualized, standards-based decisions the paper elsewhere says algorithms cannot capture. The paper's own Figure 6 shows a pair of family-assault cases where the judge differentiated on 'intensity of the crimes,' a factor not in the structured data; the paper classifies this as noise. If those cases are a legitimate mode of decision-making rather than data error, then deleting them changes the population to which the headline applies. The claim then becomes: judges behave algorithmically in the cases where their decisions are algorithmically explainable, which is circular and not the stated research question. The reader's weakest_assumption identified this same issue; my read agrees. The concern does not invalidate the paper's secondary contributions: the finding that judges are inconsistent with each other (pairwise loss, rule mining, triplet analysis) is largely independent of the filtering, and the variable-importance methodology is a useful contribution. But the central 'algorithmic' conclusion needs to be re-estimated on the full dataset and, ideally, on each unexplainable category separately. That is why the appropriate verdict is conditional acceptance: the framework is valuable, but the headline claim should not be published in its current form without a full-data robustness check.","tokens_in":21067,"tokens_out":5181,"duration_ms":44989,"concrete_test":"Re-run the identical TreeFARMS/GOSDT pipeline (depth <=5, regularization 0.001, balanced classes) on the full 22,361 cases without deleting unexplainable cases, and report per-judge balanced accuracy and the number of judges reaching the 85% threshold. Then, using the 100 manually reviewed cases (or a new annotated sample), split the excluded cases into the three reported categories and repeat the analysis including only category-(c) cases. If the full-data accuracy or the 85% count drops materially, the headline conclusion is an artifact of the exclusion; if it does not, the exclusion was benign. Report also 5-fold cross-validated accuracy for the filtered and unfiltered models, since the current accuracies appear to be in-sample.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central conclusion depends on excluding 'unexplainable cases' before fitting the GOSDT trees (section 'When do judges not follow any reasonable algorithm?', followed by 'we will remove all the unexplainable cases'). Unexplainable cases are defined as cases misclassified by every model in the Rashomon set. Manual review of 100 such cases found three causes, and the third is 'case-specific qualitative details that judges referenced but were absent from structured data' (e.g., crime intensity, Figure 6). The paper categorizes all three as 'noise' and removes them. But category (c) is not a measurement artifact; it is the judge using information outside the feature set, which is precisely the standards-based behavior the paper claims to distinguish from algorithmic rule-following. Calling it noise and deleting it makes the subsequent claim '17 of 21 judges have at least 85% of decisions captured by a small tree' true by construction only for the subset of cases representable in the feature space. The claim 'judges generally behave algorithmically' is therefore underdetermined: the analysis never measures how algorithmic judges are on the cases that motivated the study. Since unexplainable cases are 2-33% per judge (about 18% overall), and the paper does not report the fraction of that 18% attributable to category (c) versus form errors or out-of-county history, the size of the bias is unknown.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether 21 magistrate judges in Harris County, Texas behave \"algorithmically\" in misdemeanor bail hearings, operationalizing algorithmic behavior as predictability by sparse decision trees (depth ≤ 5) within a Rashomon set. The authors generate Rashomon sets with TreeFARMS, identify \"unexplainable\" cases that no near-optimal tree predicts correctly, manually review 100 such cases, classify them as noise (form errors, out-of-county criminal history, case-specific qualitative details), and remove them. On the filtered data, they fit GOSDT trees, report that 17 of 21 judges have at least 85% of decisions captured by a tree, analyze variable importance, and run cross-judge consistency, rule-mining, and triplet experiments. The paper concludes that judges generally behave algorithmically but follow different algorithms, leading to inter-judge inconsistency.","tokens_in":21360,"tokens_out":5200,"duration_ms":43044,"significance":"If the conclusions held on the full data, this would be a significant empirical contribution to the rules-versus-standards debate and to the literature on judicial decision-making under the O'Donnell Consent Decree. The paper's methodological strengths include the use of Rashomon sets to avoid single-model artifacts, public code, a fully documented data-processing pipeline, and a manual audit of unexplainable cases. However, the central empirical claim currently depends on the contested removal of 18% of cases, including cases where judges used individualized information absent from the structured data; this is precisely the standards-based behavior the study aims to distinguish from algorithmic rule-following. As a result, the headline claim is underdetermined as written.","major_comments":[{"comment":"The paper's definition of 'unexplainable' is not limited to noise: the third category identified in the manual review (case-specific qualitative details such as crime intensity, Figure 6) consists of judges using information outside the structured feature set, which is exactly the standards-based behavior the research question is designed to detect. Removing these cases before fitting the GOSDT trees means the reported accuracy (17 of 21 judges with ≥85% of decisions explained) is computed on the subset of cases that are representable in the feature space, not on the full set of decisions the paper claims to characterize. The abstract's conclusion that 'judges generally behave algorithmically' is therefore not supported unless the paper reports the fraction of the 18% of unexplainable cases attributable to each of the three categories and shows that the conclusion is robust to including category (c) cases, e.g., by fitting trees on the full data or treating category (c) as a distinct outcome.","section":"'When do judges not follow any reasonable algorithm?' and subsequent filtering step"},{"comment":"The manual review of 100 unexplainable cases is used to justify the claim that 'all of them were unexplainable due to noise,' but the paper does not describe how the sample was selected or how the three categories were coded. Without a random, stratified sampling scheme and a documented classification protocol, the extrapolation from 100 cases to the roughly 4,000 unexplainable cases is not statistically grounded. The paper should report the sampling method, the per-judge and per-category breakdown of the sample, and inter-rater reliability.","section":"Manual review of 100 unexplainable cases"},{"comment":"Appendix D states that the Rashomon set is generated with a bound multiplier of 0.01 (1% tolerance). The per-judge unexplainable fraction (2%-33%) and hence the size of the filtered subset are direct functions of this tolerance; a different tolerance would change which cases are excluded and could change the 17-of-21 result. The paper should include a sensitivity analysis over the Rashomon tolerance to demonstrate that the headline accuracy is not an artifact of this parameter.","section":"Appendix D: Rashomon set bound multiplier"}],"minor_comments":[{"comment":"The caption of Table 3 says 'Judge 4 (left) and Judge 16 (right)' but the text refers to Judges 4 and 6; the mismatch should be corrected.","section":"Table 3 and surrounding text"},{"comment":"The paper contains several typos, e.g., 'Harrasement' in Table 3, 'unexplinaable' in Appendix G, 'signifigance' and 'differnt' in Appendix H, and 'comnplex' in Appendix K; a careful proofread is needed.","section":"Throughout"},{"comment":"The abstract's statement that judges 'generally behave algorithmically' should be qualified to refer to the cases that remain after the proposed filtering, until the robustness analysis is provided.","section":"Abstract"},{"comment":"The main text refers to 'Figure 28' for additional pairwise loss comparisons, but the figure numbering in the appendix is inconsistent with the text; please check all cross-references.","section":"Appendix L and main text references"},{"comment":"Appendix H reports that some case types (e.g., ICE holds, indigency flags) differ across judges; the paper should explain more directly how this affects the interpretation of the disagreement rules in Table 4.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The paper is from a well-known group in interpretable ML and addresses a timely policy-relevant question. The main issue is the contested exclusion of unexplainable cases; this is fixable with additional analysis and reframing. I recommend major revision rather than rejection. The authors should also consider toning down the abstract until the sensitivity analyses are in place."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the paper asks a good question and builds a novel tool for it. Defining each judge's Rashomon set of near-optimal sparse trees, then using it to find cases no good algorithm can explain, is a real methodological step forward. So is averaging variable importance over the whole set rather than over one model. The inconsistency results are robust and important: judge-specific trees transfer poorly across judges, and the triplet analysis shows self-consistency around 76% versus cross-judge agreement near 55%, which is close to chance. Those findings deserve attention.\n\nThe soft spot is the unexplainable-cases filter, and it lands on the central claim. The paper removes about 18% of all decisions (2–33% per judge) because no tree in the Rashomon set predicts them, then calls all of those 'noise.' Manual review of 100 cases found three causes. Two are genuine measurement problems: form errors and out-of-county criminal history missing from the dataset. The third is judges using case-specific qualitative details—such as crime intensity—that are not in the structured data. That is not noise. That is the very standards-based, individualized judgment the paper claims to be separating from algorithmic rule-following. Removing those cases before fitting trees, and then reporting that 17 of 21 judges have at least 85% of decisions captured by a small tree, makes the headline true on the subset of cases the feature space can represent. The paper does not report what fraction of the removed 18% falls into category (c), so we cannot tell how badly this distorts the algorithmic claim.\n\nTo be clear, I don't think this is fatal to the whole paper. The framework, the inconsistency findings, and the public code are solid. But the headline needs revision. Two fixes matter most: report results on the full dataset without the exclusion, and split the unexplainable cases into data errors versus unmeasured-but-real judging. Also, the reported tree accuracies are in-sample; out-of-sample performance would be a stronger test.\n\nThis deserves a serious referee between peer review and desk reject. I would send it to review, with a clear ask: re-analyze without the loaded exclusion or substantially rework the claim. Whoever reads it will learn something either way.","headline":"A genuinely useful Rashomon-set framework for studying judicial consistency, but the headline that judges 'behave algorithmically' rests on removing the very cases where they used unmeasured, substantive judgment.","tokens_in":21913,"tokens_out":2123,"would_cite":true,"duration_ms":22963,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most Harris County magistrates decide bail by small, predictable rules; the rules differ by judge.","keywords":["judicial decision-making","bail hearings","interpretable machine learning","sparse decision trees","Rashomon set","judge inconsistency","rules versus standards","pretrial release"],"falsifier":"A direct falsifier: take the removed unexplainable cases, obtain the hearing transcripts, and code the qualitative details the judges cited (severity of injury, out-of-county history, defendant statements); if those details predict the bond ruling consistently across judges, the 18% are structured judgment, not noise, and the central claim fails. A simpler version is to rerun the entire pipeline without deleting unexplainable cases and check whether most judges' depth-five trees still capture at least 85% of decisions.","tokens_in":20852,"feed_emoji":"⚖️","tokens_out":10722,"duration_ms":91482,"temperature":0.7,"pith_summary":"The paper asks whether magistrate judges in Harris County, Texas, decide misdemeanor bail cases the way a simple algorithm would, rather than debating whether algorithms should replace judges. Using more than 22,000 hearings before 21 magistrates, it finds that most judges' decisions can be captured by small decision trees of depth at most five, with 17 of 21 judges having at least 85% of their decisions matched by such a tree. The same analysis shows that judges do not all follow the same tree: models trained on one judge perform substantially worse on another, and judges agree with each other on similar cases only about 55% of the time, near chance. About 18% of cases are \"unexplainable\" by any good tree, and a manual review attributes these to data-entry errors, criminal history outside Harris County, or case-specific qualitative details such as crime intensity. The paper's conclusion is a paradox: judges behave algorithmically, but each behaves like a different algorithm, so outcomes depend heavily on which judge is assigned.","feed_headline":"Harris County bail judges act like algorithms, just different ones","feed_subtitle":"Small trees capture 85% of rulings for most magistrates, yet similar cases agree only ~55% across judges.","key_machinery":"The load-bearing object is the Rashomon set of sparse decision trees: for each judge, the complete collection of near-optimal trees of depth at most five, kept within a 1% tolerance of the best objective. This set does three jobs. First, it defines \"unexplainable\" cases as those every tree in the set gets wrong, which is how the paper isolates noise and standards-driven outliers. Second, it supports variable importance averaged over many equally good models instead of one model, avoiding the common flaw of attributing importance to a single fitted tree. Third, it supplies the source models for cross-judge loss distributions and for frequent-pattern rule mining that identifies disagreement conditions such as defendants aged 24 or younger. A second mechanism, globally optimal sparse decision-tree fitting, selects one representative shallow tree per judge to quantify how algorithm-like each judge is.","core_discovery":"The paper's central discovery is that judicial behavior at these bail hearings is largely rule-like and formulaic, but the formula is judge-specific. The authors define \"unexplainable\" cases as those that no near-optimal shallow decision tree (a tree of depth at most five within the Rashomon set) can predict correctly, identify about 18% of cases this way, and find by manual review of a sample that all such cases trace to data-entry errors, out-of-county criminal history missing from the dataset, or case-specific qualitative details the judge heard but the structured data do not record. After removing those cases, globally optimal sparse trees reconstruct 75–100% of each judge's decisions, and for 17 of 21 judges at least 85% of rulings are captured. Across judges, the same variables (age, warrant count, and rule-based flags) recur but with different weights and additional variables; when a model trained on one judge is applied to another, accuracy drops sharply, and on matched similar cases judges agree only about 55% of the time, close to chance. The paper concludes that judges behave algorithmically yet inconsistently: the justice system's outcomes depend heavily on which magistrate is assigned.","pith_inferences":["Because the paper operationalizes \"algorithmic\" as depth at most five with an 85% match, the headline conclusion is partly definitional; rerunning with depth three or a 90% threshold would reclassify some judges, and we predict the ordering of judges by algorithmic-ness would remain stable even if the \"generally\" qualifier would shift.","The near-chance cross-judge agreement implies an audit design: match cases on all structured features, compare release outcomes across magistrates, and test whether judge identity remains predictive; if it does, assignment is effectively a lottery.","The unexplainable-case detector could be inverted into a real-time monitoring tool that flags a ruling falling outside a judge's own Rashomon set before detention, prompting a second look at exactly the cases where standards rather than rules may be operating.","The variable-importance result that race matters for some judges suggests a concrete next test: whether the mined disagreement rules involving age, family assault, and prior warrants differentially affect defendants by race; the paper does not report that demographic breakdown."],"forward_implications":["If judges are broadly rule-following, improving judicial behavior can be framed as improving the rules: better data and standardized guidelines can be targeted rather than treated as inscrutable discretion.","Cases that no reasonable tree explains are where individualized standards actually operate; focusing oversight there, rather than on routine cases, can surface data errors, missing criminal history, and case-specific factors.","Because each judge's algorithm differs, aggregating judicial behavior into a single model is misleading; models must be judge-specific, and alignment interventions should target judge pairs with strong disagreement rules.","Inter-judge agreement near chance on similar cases means the assignment of a magistrate materially changes detention outcomes, a direct challenge to consistency as a due-process value.","Feature importance averaged over the Rashomon set is more robust than single-model importance; applying it to judges can reveal which demographic characteristics, such as race, each judge relies on."],"supporting_citations":[{"why":"Supplies the TreeFARMS algorithm that enumerates each judge's Rashomon set of sparse decision trees, the basis for detecting unexplainable cases.","marker":"(Xin et al. 2022)"},{"why":"Supplies GOSDT, the globally optimal sparse decision-tree method used to produce each judge's representative algorithm.","marker":"(Lin et al. 2020)"},{"why":"Provides the rationale and method for averaging variable importance over an entire class of good models rather than a single model.","marker":"(Fisher, Rudin, and Dominici 2019)"},{"why":"Establishes that judge-specific preferences create variation in bail outcomes, the prior result this paper's judge-consistency analysis builds on.","marker":"(Arnold, Dobbie, and Hull 2022)"},{"why":"Shows UK bail decisions can be captured by fast-and-frugal decision trees, the prior evidence that judicial behavior is heuristic and rule-like.","marker":"(Dhami and Ayton 2001)"},{"why":"Contextualizes the human-versus-algorithm comparison in pretrial release that this paper deliberately reframes as a question about judges' internal structure.","marker":"(Kleinberg et al. 2018)"},{"why":"Documents the downstream consequences of Harris County misdemeanor pretrial detention and the reform context for the post-2020 dataset.","marker":"(Heaton, Mayson, and Stevenson 2017)"},{"why":"Supplies the frequent-pattern growth algorithm used to mine the disagreement rules that characterize when judge pairs diverge.","marker":"(Han, Pei, and Yin 2000)"},{"why":"Supplies the Threshold Guessing Binarizer that converts continuous features into binary splits for the decision-tree models.","marker":"(McTavish et al. 2022)"},{"why":"Provides the monitoring context and data infrastructure for the Harris County bail system under the consent decree.","marker":"(Garrett et al. 2024)"}],"fun_headline_variants":["Judges act like algorithms—but each has a different one","Predictable judges, inconsistent court outcomes","Bail judges: formulaic individually, erratic collectively","Small decision trees capture judges, but not agreement","Judge algorithms vary widely, causing unequal treatment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the roughly 18% of cases no good tree can explain are mostly noise — form errors, missing out-of-county history, or unrecorded case-specific details — rather than a systematic standards-based mode of judging; if those cases represent real, consistent judicial reasoning, then the conclusion that judges generally behave algorithmically is an artifact of deleting them.","fun_headline_variants_meta":{"raw":{"variants":["Judges act like algorithms—but each has a different one","Predictable judges, inconsistent court outcomes","Bail judges: formulaic individually, erratic collectively","Small decision trees capture judges, but not agreement","Judge algorithms vary widely, causing unequal treatment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1680,"prompt_tokens":1039,"completion_tokens":641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":569}},"tokens_in":655,"tokens_out":641,"duration_ms":7138,"temperature":1.0,"reasoning_tokens":569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:20:52.226665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier: take the removed unexplainable cases, obtain the hearing transcripts, and code the qualitative details the judges cited (severity of injury, out-of-county history, defendant statements); if those details predict the bond ruling consistently across judges, the 18% are structured judgment, not noise, and the central claim fails. A simpler version is to rerun the entire pipeline without deleting unexplainable cases and check whether most judges' depth-five trees still capture at least 85% of decisions.","supporting_citations":[{"cited_title":"American Economic Review , Volume =","cited_arxiv_id":null,"evidence_quote":"Establishes that judge-specific preferences create variation in bail outcomes, the prior result this paper's judge-consistency analysis builds on."},{"cited_title":"2017 , journal =","cited_arxiv_id":null,"evidence_quote":"Documents the downstream consequences of Harris County misdemeanor pretrial detention and the reform context for the post-2020 dataset."},{"cited_title":"2000 , issue_date =","cited_arxiv_id":null,"evidence_quote":"Supplies the frequent-pattern growth algorithm used to mine the disagreement rules that characterize when judge pairs diverge."}],"review_version":1}