{"id":"c02e6625-6603-4e92-83d8-35ac6d84eee9","arxiv_id":"2503.03888","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A fine-tuned open language model detected thousands of racial covenants in Santa Clara County deed records at low cost, leading to an estimate that one in four county properties was racially restricted by 1950.","lead":"Researchers fine-tuned an open AI language model to scan 5.2 million property deed pages in Santa Clara County and flag racial covenants, with high reported accuracy and at a tiny fraction of the cost of manual review or commercial AI. The work gives counties a cheaper path to comply with California's 2021 law and produces new historical evidence on how widespread housing discrimination was.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation set for headline precision/recall is keyword-enriched and pre-1940 only; deployment recall on the 1902-1980 corpus is unmeasured, so the 7,500-deed count and one-in-four estimate rest on unvalidated transfer.","rationale":"The central quantitative claims are the detector's accuracy and the counts derived from applying it. The weakest point is the link between the evaluation numbers and the deployment population. The evaluation set is not a random sample: it is a keyword-enriched, model-assisted annotation pool, and within Santa Clara it is restricted to pre-1940 documents. This makes the headline precision/recall a measure of performance on an easy, narrow slice rather than on the full 1902-1980 corpus. The paper is unusually transparent about this in Section 6: it reports later-period false positives and a post hoc threshold, and it does provide a separate deployment-precision estimate from 200 sampled positives. But no equivalent deployment-recall estimate is provided, and recall is the quantity that determines whether the 7,500 count and the \"one in four\" prevalence are undercounts. The Section 8 prevalence workflow adds further assumptions, including neighborhood-covenant detection, LLM lot extraction, and deduplication, but if the base detector misses later-period or atypical covenants, those upstream counts are biased regardless of the care taken downstream. A stratified random-sample audit would settle the transfer. This does not invalidate the application: the model release, the attorney-confirmed subset, and the cost comparison are real contributions. But the headline accuracy claims and the historical counts should be presented as conditional on validation on a representative sample, which is exactly what the conditional verdict requires.","tokens_in":27455,"tokens_out":6127,"duration_ms":59337,"concrete_test":"Draw an independent random sample of roughly 1,000 pages, stratified by recording era (1902-1940, 1941-1960, 1961-1980), from the full 5.2-million-page corpus, outside the keyword/model-selection loop. Have county counsel or trained annotators label each page for presence of a racially restrictive covenant, then run the deployed detector with the 75% confidence threshold and fair-housing filter. Compute precision and recall overall and per stratum. If recall on post-1940 strata falls materially below 0.99, or precision on the full-corpus sample is below the deployed 96.4-99.7% interval, the 7,500-deed count and 1950 prevalence estimate should be revised with period-specific error bounds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 builds the annotation pool by first selecting 3,000 pages using keywords that \"almost certainly indicate the presence of a racial covenant,\" then adding model predictions and hard negatives; the 739-page evaluation set is drawn from this pool, so roughly 70% of pages contain covenants versus fewer than 2 per 1,000 deeds in the full corpus. Section 6 then concedes that the Santa Clara evaluation documents were \"exclusively pre-1940s,\" while deployment covers 1902-1980 and surfaced fair-housing false positives that required a post hoc 75% confidence threshold and a \"fair housing\" filter. Consequently, the reported precision 1.0 / recall 0.99 is not an estimate for a probability sample of the target corpus. Deployment precision was separately estimated from 200 sampled positives (96.4-99.7%), but deployment recall was never measured. The downstream claims about 7,500 covenant deeds and one-in-four properties covered by 1950 inherit whatever recall error exists on later-period or atypical documents, and the paper gives no bound on that error.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes an academic-government partnership that built an OCR plus language-model pipeline to detect racially restrictive covenants in Santa Clara County's pre-1980 deed records. The authors fine-tune Mistral 7B with LoRA and report page-level precision 1.000 and recall 0.994 on a held-out set of 739 annotated pages, compare against keyword and GPT-3.5 baselines, and estimate that running the pipeline over 5.2 million pages costs $258 and saves 86,500 person-hours of manual review. Using the detector's outputs, geolocation, and census data, the paper estimates roughly 7,500 covenant deeds and that one in four 1950 properties were covered by racial covenants, and it derives historical, geographic, and developer-level findings. The paper also documents the integration of the model into the county's AB 1466 redaction workflow and states that the model and review interface will be released.","tokens_in":27790,"tokens_out":7243,"duration_ms":71265,"significance":"If the performance claims held up, this would be a significant contribution: a concrete, deployed demonstration that fine-tuned open LLMs can reduce the cost of legal-record review by orders of magnitude, plus a population-scale quantitative map of racial covenants. The partnership details, the random-sample deployment precision check (200 positives, 96.4-99.7% CI), the manual verification of geolocations on a 50-document sample, and the transparent discussion of deployment false positives are genuine strengths. The manuscript is also careful in Section 8 to list directional biases in the prevalence estimate. However, the central evaluation claim is currently not supported by the evidence as presented, because the evaluation set does not represent the deployment corpus and deployment recall is unmeasured; the historical and prevalence findings inherit that gap. The contribution remains valuable, and the gap is addressable with additional validation rather than being fatal to the approach.","major_comments":[{"comment":"The headline evaluation is not representative of the deployment population. The 739-page evaluation set described in Section 5.2 is drawn from the annotation pool constructed in Section 3.3, which began with 3,000 keyword-selected pages and then added model predictions and hard negatives; about 70% of the evaluation pages contain covenants, whereas the full corpus has fewer than 2 per 1,000. Section 6 further states that the Santa Clara County evaluation documents were 'exclusively pre-1940s' while the deployed system processes 5.2 million pages from 1902 to 1980, and that later-period documents produced fair-housing false positives requiring a post hoc confidence threshold and a 'fair housing' filter. The deployment precision estimate from 200 sampled positives (96.4-99.7%) is a useful check, but deployment recall was never measured. Therefore the abstract's 'precision of 1.0 and a recall of 0.99' and the downstream 7,500-deed count are not supported as estimates for the full corpus.","section":"5.2 / Table 2 and Section 6"},{"comment":"The 75% confidence threshold was chosen on the same evaluation set used to report the headline metrics. Section 4.2 says the threshold was 'empirically determined' and that the Section 5.1 results are computed at that threshold; because the same 739 pages were used for threshold selection and for the reported precision/recall, the numbers in Table 2 are selection-optimized. Section 6 then changes the operating point again at deployment by adding the 75% threshold and a 'fair housing' filter, without measuring recall on a sample from the deployment distribution. The paper should either use a separate validation set for threshold selection or present the metrics as conditional on the tuning set, and it should add a deployment recall estimate on a stratified random sample.","section":"4.2 and Section 6"},{"comment":"The estimate that one in four properties were covered by a racial covenant in 1950 inherits the unmeasured recall error described above. The count of 24,522 lots is built from detector outputs for neighborhood-wide declarations (412 manually confirmed), multi-lot deeds (1,293, of which only 20 units were verified according to the footnote), and single-lot deeds; no sensitivity analysis varies the detector's recall or the classification of multi-lot deeds. The manuscript lists directional biases that could move the estimate up or down, but a quantitative lower/upper bound or confidence interval is needed for the headline prevalence claim.","section":"8"},{"comment":"The historical periodization, including the 'near 75% decrease' after Shelley and the persistence of covenants after 1948, compares detector counts across decades. Because detection recall is unvalidated outside the pre-1940s Santa Clara evaluation documents, and later-period documents are known to contain different language (e.g., fair-housing covenants), the observed drop could partly reflect a decline in detection sensitivity rather than a decline in covenant usage. A stratified recall evaluation by decade is required before these historical conclusions are stated as results.","section":"7.3"}],"minor_comments":[{"comment":"The text says the external data came from 'seven counties' but then lists six counties (Bexar, Cuyahoga, Denton, Franklin, Hidalgo, and Lawrence); please correct the count or add the missing county.","section":"3.2"},{"comment":"Appendix E states that the total cost for an off-the-shelf LLM would be 'roughly $30,000,' but Table 1 reports $13,634 for GPT-3.5 few-shot and $47,944 for GPT-4 Turbo; please reconcile these numbers.","section":"Appendix E"},{"comment":"The phrase 'one in four properties the county were covered' is missing the word 'in' in both Section 1 and Section 7's opening; also 'Mistral 7B parameter open source model' should read 'Mistral 7B-parameter open source model.'","section":"1 and 7"},{"comment":"The column header 'Native White' is ambiguous; the table appears to report 'White' and 'Native American' as separate columns, and the 1980 'Others' spike is explained only in the caption; please clarify the column labels in the header.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"This is a strong applied contribution and the paper is unusually transparent about deployment false positives and limitations. My main concern is the mismatch between the evaluation design and the deployment claims: the evaluation set is keyword-enriched and pre-1940s for Santa Clara County, while the deployment covers 1902-1980, and deployment recall is never measured. This is fixable within the manuscript's scope by adding a stratified deployment recall study and recalibrating the abstract and Section 5 claims. No concerns about novelty or authorship integrity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper delivers a real, deployed system and a useful first county-scale historical estimate, but the headline precision/recall is measured on an evaluation set that does not represent the deployment corpus, and deployment recall is unmeasured. The weaknesses are real, but the paper names most of them in Sections 6 and 9; the main fix is to re-frame the claims and add a proper probability sample.\n\nWhat is genuinely new: they built a LoRA-finetuned Mistral model, ran it over Santa Clara County's 5.2 million pre-1980 deed pages, integrated it with the county recorder's workflow, and got attorney review for over 4,500 redactions. The cost data is concrete: about $258 in GPU rental, six days, versus ~$13.6K for few-shot GPT-3.5 and over $1.4M in manual labor. Releasing the model is a real contribution for the many jurisdictions under AB 1466. The historical findings—roughly 7,500 covenant-bearing deeds, one in four properties by 1950, ten developers responsible for a third of covenants, a city-owned cemetery selling restricted burial plots—are interesting and come from a scale no volunteer project has reached.\n\nNow the soft spots, in proportion. The evaluation set is built from a keyword-enriched pool: about 70% of pages contain covenants, while the full corpus has fewer than 2 per 1,000 deeds. That alone makes the headline precision 1.0 / recall 0.99 untransferable. More importantly, Section 6 concedes the Santa Clara evaluation documents were exclusively pre-1940s, while deployment covers 1902–1980; the later-period false positives they describe (fair-housing covenants) are exactly the kind of distribution shift that can break a classifier. Deployment precision was re-estimated on 200 sampled positives (96.4–99.7% CI), which is good, but deployment recall was never measured. The 7,500-deed count and the one-in-four estimate therefore rest on an unvalidated assumption that recall is stable across time and atypical documents. The confidence threshold was also tuned on the evaluation set, which further inflates the reported numbers.\n\nThat said, the paper is honest about many of these issues: Section 6 reports the deployment false positives plainly, Section 8 lists the assumptions behind the prevalence estimate, and Section 9 acknowledges imperfect recall. The presentation problem is that the abstract and introduction repeat the precision/recall without the caveat, and the historical counts are treated as facts rather than estimates with an unknown error bar.\n\nWho should read this: applied ML and public-administration researchers, county recorders, and historians of housing discrimination will all get value. It deserves a serious referee. My recommendation: accept conditional on a probability-sample evaluation across the full 1902–1980 corpus, a deployment recall audit (even a modest one), and confidence intervals or sensitivity bounds on the historical counts. The core contribution is solid; the numbers just need to match the evidence.","headline":"Genuinely useful deployment paper, but the headline metrics come from a non-representative eval set and deployment recall is unmeasured; the contribution survives, the numbers need reframing.","tokens_in":28205,"tokens_out":3494,"would_cite":true,"duration_ms":34372,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A $258 AI run finds 7,500 racial covenants in Santa Clara County deeds.","keywords":["racial covenants","large language models","property deeds","Santa Clara County","LoRA fine-tuning","housing discrimination","AB 1466","legal reform"],"falsifier":"Take a period-stratified random sample of 300 deed pages from 1940 to 1980 in Santa Clara County, have two independent legal annotators label them for racial covenants, and run the released model at the 75 percent confidence threshold with the fair-housing filter. If precision on that sample falls below 95 percent, or recall falls below the claimed 0.99, the deployment-time accuracy claim is falsified.","tokens_in":27259,"feed_emoji":"📜","tokens_out":7737,"duration_ms":67056,"temperature":0.7,"pith_summary":"This paper claims that a fine-tuned open large language model can find racially restrictive covenants in historical property deeds at a scale and cost that make statewide legal reform feasible. Working with Santa Clara County's Clerk-Recorder's Office, the authors built an OCR-to-text pipeline and finetuned a Mistral 7B model with LoRA, reporting precision of 1.0 and recall of 0.99 on a held-out evaluation suite of 739 deed pages. Running the model over 5.2 million pre-1980 deed pages cost about $258 in GPU rental, identified roughly 7,500 covenant deeds, and estimated savings of about 86,500 person-hours of manual review. The paper also uses geolocated results to document the history of housing discrimination, estimating that one in four Santa Clara County properties was subject to a racial covenant by 1950. If the accuracy and cost figures hold, the approach offers a template for the hundreds of jurisdictions required to scan their deed records.","feed_headline":"$258 AI run finds 7,500 racial covenants in county deeds","feed_subtitle":"A fine-tuned open model scans 5.2 million deed pages and replaces roughly 86,500 hours of human review.","key_machinery":"The load-bearing object is a LoRA-finetuned Mistral 7B language model, where LoRA (low-rank adaptation) updates a small set of weight matrices rather than the full model. The model is fed OCR text from deed pages: docTR converts page images to text, the model classifies each page and, when positive, returns the exact covenant passage, and a confidence score based on the softmax of the 'yes' and 'no' tokens, thresholded at 75 percent, filters low-confidence predictions. A separate geolocation pipeline extracts map names and book-and-page references from deed text and matches them to county surveyor maps, which is what turns a document classifier into a map of housing discrimination.","core_discovery":"The central discovery is that a 7-billion-parameter open language model, finetuned with low-rank adaptation on 3,801 annotated deed pages, can outperform keyword search, fuzzy matching, and zero- or few-shot prompting of an off-the-shelf closed model in detecting racial covenants. On the paper's own evaluation set it achieves perfect precision and 99.4 percent recall, with a span-level BLEU of 0.932, and it runs for a fraction of the cost of a proprietary API. The authors go on to show that the model's outputs can be integrated into a legally required human review workflow, with county counsel confirming more than 4,500 of the identified covenants, and that the resulting dataset reveals sharp geographic clustering and the outsized role of about ten developers. Their population-level estimate is that by 1950 one in four properties in the county was encumbered by a racial covenant.","pith_inferences":["Editorial inference: the same pipeline should extend to other discriminatory restrictions, such as religion, family status, or national origin, because the paper already reports surfacing a family-status covenant and few-shot prompting or further finetuning would likely retrieve those with the same architecture.","Editorial inference: a period-stratified audit of 1960s and 1970s deeds would likely show lower precision than the headline 1.0, since the evaluation set is pre-1940s and the paper itself documents fair-housing false positives in later documents.","Editorial inference: OCR quality, not model quality, is the binding constraint for older records; the paper excludes pre-1902 handwritten deeds for lack of effective OCR, so better handwriting recognition would extend coverage backward in time.","Editorial inference: the approximately $258 cost and under-2-percent comparison are time-bound to current GPU rental and API prices, though the qualitative advantage of open models at this scale should persist."],"forward_implications":["Counties facing AB 1466-style mandates can complete a full deed sweep without years of manual review; the paper estimates roughly 86,500 person-hours saved for Santa Clara County's 5.2 million pre-1980 pages.","Open-weight models make comprehensive deed scanning economical; the paper's comparison puts the full county run at $258, under 2 percent of the cost of an off-the-shelf closed model.","The roughly 7,500 identified covenant deeds, once geolocated, produce the paper's historical findings: one in four properties covered by 1950, ten developers responsible for about a third of covenants, and continued use after Shelley v. Kraemer.","Releasing the fine-tuned model and review interface gives other jurisdictions a starting point, provided they validate on their own records as the paper recommends.","The county's redaction workflow retains an unredacted registry, so removing discriminatory language from active records does not erase the historical evidence."],"supporting_citations":[{"why":"Supplies the Mistral 7B base model that the authors finetune; the detector's performance starts from this architecture.","marker":"Jiang et al., 2023"},{"why":"Supplies the LoRA low-rank adaptation method used to finetune the base model on labeled deed pages.","marker":"Hu et al., 2021"},{"why":"Supplies the docTR OCR library that converts deed page images into text, the input on which every detector is evaluated.","marker":"Mindee, 2021"},{"why":"Provides the AB 1466 statutory mandate that defines the operational problem and requires county counsel review of redactions.","marker":"California State Legislature, 2021"},{"why":"Supplies the historical account of racial covenants and their post-Shelley persistence against which the paper's empirical findings are set.","marker":"Brooks and Rose, 2013"},{"why":"Supplies the BLEU metric used to evaluate how precisely the model reproduces the racial covenant text span.","marker":"Papineni et al., 2002"}],"fun_headline_variants":["Open AI model scans county deeds to find racial covenants","AI maps a century of racist housing clauses in Santa Clara","86,500 hours saved by AI redaction of racial covenants","One in four county homes had racial covenants by 1950","County deploys open LLM to strike racist deed clauses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation set that produced the precision-1.0 and recall-0.99 numbers contains only pre-1940s Santa Clara County deeds, while the deployed scan covers 1902 to 1980, so the reported accuracy may not hold for the later documents unless the confidence threshold and fair-housing filter fix the drift.","fun_headline_variants_meta":{"raw":{"variants":["Open AI model scans county deeds to find racial covenants","AI maps a century of racist housing clauses in Santa Clara","86,500 hours saved by AI redaction of racial covenants","One in four county homes had racial covenants by 1950","County deploys open LLM to strike racist deed clauses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3161,"prompt_tokens":1030,"completion_tokens":2131,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":2049}},"tokens_in":646,"tokens_out":2131,"duration_ms":13767,"temperature":1.0,"reasoning_tokens":2049,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:29:22.142500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a period-stratified random sample of 300 deed pages from 1940 to 1980 in Santa Clara County, have two independent legal annotators label them for racial covenants, and run the released model at the 75 percent confidence threshold with the fair-housing filter. If precision on that sample falls below 95 percent, or recall falls below the claimed 0.99, the deployment-time accuracy claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the docTR OCR library that converts deed page images into text, the input on which every detector is evaluated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AB 1466 statutory mandate that defines the operational problem and requires county counsel review of redactions."}],"review_version":1}