{"id":"041ebe1c-4012-4565-a355-bc4b90faa6e2","arxiv_id":"2605.24452","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Transformer models for legal judgment prediction show up to 27.2 pp macro-F1 forward degradation across three Ukrainian court epochs, with asymmetric transfer, reduced degradation from legal pretraining, and gains from chronological continual learning.","lead":"The paper tests whether legal language stays the same over time by training transformer models on Ukrainian court decisions from three periods split by major geopolitical events and measuring performance drops across periods. If the reported degradation holds, legal NLP systems will need time-aware training and evaluation to remain useful in practice.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Epoch performance gaps may reflect shifts in case-type distribution or court procedures rather than language drift.","rationale":"The reader's weakest_assumption already isolates exactly this confound. Because the abstract-only review could not inspect the methods or supplementary tables for distribution checks, the concern remains load-bearing; the full-text description does not resolve it.","tokens_in":1793,"tokens_out":322,"duration_ms":17209,"concrete_test":"Tabulate the proportion of each judgment category and mean token length across the three epochs on the released 428K dataset; if any category shifts by >10 percentage points, recompute the 3x3 cross-temporal matrices after inverse-propensity reweighting to the pre-war marginals and report whether the 27.2 pp gap remains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The three epochs are defined solely by geopolitical cut-points (2008-2013, 2014-2021, 2022-2026). The central claim attributes the 27.2 pp macro-F1 forward drop to temporal evolution in legal language. No section in the provided text reports the marginal distribution of judgment categories (civil/criminal/administrative), court instance, or outcome labels per epoch, nor any reweighting, stratification, or propensity matching to hold case composition fixed. If these distributions differ systematically (e.g., higher proportion of war-related criminal cases post-2022), the observed degradation is consistent with a change in the data-generating process rather than non-stationarity of the language model itself.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that legal language in Ukrainian court decisions is non-stationary across three geopolitically defined epochs (pre-war 2008-2013, hybrid war 2014-2021, full-scale invasion 2022-2026). It supports this by training four transformer models (XLM-RoBERTa base/large and legal variants) on one epoch and evaluating on all three to produce a 3x3 cross-temporal macro-F1 matrix, reporting up to 27.2 pp forward degradation, asymmetric backward transfer, reduced degradation with legal pretraining, elimination of forgetting via chronological continual learning, and no mitigation from Swiss cross-jurisdictional pretraining. A 428K-decision dataset is released publicly.","tokens_in":1947,"tokens_out":623,"duration_ms":17667,"significance":"If the performance gaps survive controls for case-type and procedural shifts, the work supplies concrete evidence that random splits in legal NLP benchmarks violate stationarity and that chronological continual learning is an effective mitigation. The public dataset release and the explicit 3x3 matrix design are clear strengths that enable future replication and extension.","major_comments":[{"comment":"Abstract and §4 (results): The central claim that the 27.2 pp macro-F1 forward drop constitutes 'temporal concept drift in legal language' is not yet supported because the manuscript reports neither the marginal distribution of judgment categories (civil/criminal/administrative), court instance, nor outcome labels per epoch, nor any reweighting, stratification, or propensity-score matching that would hold case composition fixed across the three periods.","section":"Abstract, §4"},{"comment":"§3 (experimental setup): No description is given of data filtering criteria, per-epoch label distributions, or statistical tests (e.g., McNemar or bootstrap confidence intervals) for the reported performance differences; without these, it is impossible to determine whether the numerical findings are robust to sampling variation or confounding variables.","section":"§3"},{"comment":"§4.2 (continual learning): The claim that chronological training 'eliminates catastrophic forgetting' (+1.8 to +6.2 pp retention) while reverse order causes severe forgetting is load-bearing for the practical recommendation, yet the section provides no ablation on learning-rate schedules, replay buffer size, or regularization strength that would isolate the effect of temporal ordering from other training choices.","section":"§4.2"}],"minor_comments":[{"comment":"Table 1 (dataset statistics): Epoch sizes and class balance should be reported explicitly rather than only aggregate 428K figure.","section":"Table 1"},{"comment":"Figure 2 (performance matrix): Axis labels and color scale should include exact macro-F1 values for each cell to improve readability.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below, agreeing where revisions are needed to strengthen the evidence for temporal drift and the continual learning results. All requested controls, statistics, and ablations can be added using the publicly released dataset.","responses":[{"response":"We agree that controlling for shifts in case composition is essential to isolate temporal language drift. In the revised manuscript we will report the full marginal distributions of judgment categories, court instances, and outcome labels for each epoch. We will additionally apply propensity-score matching (or stratification) on these variables and recompute the 3x3 cross-temporal matrices; any remaining degradation after matching will provide stronger support for the drift claim.","revision_made":"yes","referee_comment":"[Abstract, §4] Abstract and §4 (results): The central claim that the 27.2 pp macro-F1 forward drop constitutes 'temporal concept drift in legal language' is not yet supported because the manuscript reports neither the marginal distribution of judgment categories (civil/criminal/administrative), court instance, nor outcome labels per epoch, nor any reweighting, stratification, or propensity-score matching that would hold case composition fixed across the three periods."},{"response":"We will expand §3 to include the exact data filtering criteria applied to produce the 428K-decision corpus, the per-epoch label distributions, and statistical significance tests (bootstrap confidence intervals on macro-F1 together with McNemar tests on paired predictions). These additions will directly address concerns about robustness and confounding.","revision_made":"yes","referee_comment":"[§3] §3 (experimental setup): No description is given of data filtering criteria, per-epoch label distributions, or statistical tests (e.g., McNemar or bootstrap confidence intervals) for the reported performance differences; without these, it is impossible to determine whether the numerical findings are robust to sampling variation or confounding variables."},{"response":"We acknowledge that further ablations would strengthen the isolation of temporal ordering. The revision will add experiments varying learning-rate schedules, replay buffer sizes, and regularization strengths while keeping ordering as the sole manipulated factor. At the same time, the existing results already show a large, consistent gap between chronological and reverse-chronological training under identical hyper-parameters, supporting the recommendation for chronological continual learning.","revision_made":"partial","referee_comment":"[§4.2] §4.2 (continual learning): The claim that chronological training 'eliminates catastrophic forgetting' (+1.8 to +6.2 pp retention) while reverse order causes severe forgetting is load-bearing for the practical recommendation, yet the section provides no ablation on learning-rate schedules, replay buffer size, or regularization strength that would isolate the effect of temporal ordering from other training choices."}],"tokens_in":1611,"tokens_out":608,"duration_ms":23603,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main things to know are that this paper measures substantial forward degradation when training on earlier Ukrainian court decisions and testing on later ones, up to 27.2 macro-F1 points, and that chronological continual learning appears to retain prior performance while improving on new data.\n\nWhat it does well is release a 428k-decision public dataset split into three epochs tied to real events and run a clean 3x3 cross-temporal matrix with XLM-RoBERTa variants plus legal pretraining. The asymmetry between forward and backward transfer, the partial mitigation from legal-domain pretraining, and the directional continual learning results are concrete empirical additions beyond generic drift papers. The Swiss pretraining comparison also helps isolate the temporal issue.\n\nThe soft spot is the attribution to language evolution. The epochs are defined by geopolitical cut-points, yet the abstract gives no distributions of case categories, court instances, or outcome labels per period, nor any reweighting or matching to hold composition fixed. If the mix of civil versus criminal cases or war-related matters shifted after 2022, that alone could produce the observed drops without any change in the language itself. The stress-test concern stands on the provided text.\n\nThis is for researchers working on legal NLP or temporal adaptation in applied domains. The dataset and matrix results are usable and falsifiable, so it deserves a serious referee to check the methods and any additional controls in the full paper.\n\nI would send it to peer review.","headline":"The paper measures a 27-point forward F1 drop on a new Ukrainian legal dataset across geopolitical epochs but does not rule out case-mix shifts as the cause.","tokens_in":2427,"tokens_out":375,"would_cite":false,"duration_ms":38547,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Pre-war trained models lose up to 27.2 macro-F1 points on full-scale invasion Ukrainian court decisions due to temporal drift in legal language.","keywords":["temporal concept drift","legal judgment prediction","Ukrainian court decisions","cross-temporal evaluation","continual learning","transformer encoders","macro-F1 degradation"],"falsifier":"Re-running the 3x3 matrix after explicitly balancing case-type distributions, court levels, and procedural metadata across epochs and finding that the forward degradation disappears would falsify the claim that temporal language evolution is the cause.","tokens_in":2688,"feed_emoji":"⚖️","tokens_out":675,"duration_ms":27592,"temperature":0.7,"pith_summary":"The paper tests the common assumption in legal NLP that language is stationary by dividing 428K Ukrainian court decisions into three epochs around major geopolitical shifts and running a 3x3 cross-temporal evaluation. Four transformer models are trained on one epoch and tested on all three, revealing that forward transfer degrades sharply while backward transfer remains more stable. Legal-domain pretraining shrinks the gap somewhat, chronological continual learning fully retains old knowledge and adds substantial new performance, and Swiss pretraining raises overall scores without fixing the temporal issue. The results show that random data splits in benchmarks can mask real-world degradation when models encounter later legal language.","feed_headline":"Models lose 27 points on newer Ukrainian court rulings","feed_subtitle":"Pre-war training collapses on invasion-era cases while reverse transfer and time-ordered learning hold up better.","key_machinery":"The 3x3 cross-temporal generalization matrix obtained by training each model on one epoch and evaluating on all three epochs.","core_discovery":"The paper establishes that legal language evolves with external events, shown by a 3x3 matrix of macro-F1 scores where models trained on pre-war (2008-2013) data lose as much as 27.2 points when evaluated on full-scale invasion (2022-2026) decisions. Backward transfer from later to earlier epochs is substantially stronger, consistent with additive language change. Legal-domain variants reduce both the size and asymmetry of the degradation, while chronological continual learning retains pre-war performance (+1.8 to +6.2 points) and lifts full-scale performance (+16.5 to +19.0 points); reverse chronological order produces severe forgetting. Cross-jurisdictional pretraining improves absolute sc","pith_inferences":["The same forward-degradation pattern is likely to appear in other jurisdictions that experience major legal or political ruptures.","Benchmark construction for legal NLP should replace random splits with explicit temporal or event-based splits to measure real generalization.","Continual-learning schedules ordered by time may become a standard requirement for any deployed legal prediction system.","The public 428K-decision dataset enables direct replication and extension to other languages or model families."],"forward_implications":["Models trained on pre-war data lose up to 27.2 percentage points macro-F1 on full-scale invasion decisions.","Backward transfer is substantially more robust than forward transfer.","Legal-domain pretraining reduces both the magnitude and asymmetry of forward degradation.","Chronological continual learning eliminates catastrophic forgetting of earlier epochs while adding 16.5-19.0 points on the newest epoch."],"fun_headline_variants":["Pre-war models lose 27 points on invasion-era court rulings","Backward transfer holds while forward legal drift collapses","Chronological learning retains old knowledge lifts new scores","Legal variants cut temporal degradation asymmetry in judgments","Cross-epoch matrix shows additive change in Ukrainian decisions"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The performance gaps between epochs are produced by changes in legal language rather than by shifts in case types, court procedures, or data collection methods across the three periods.","fun_headline_variants_meta":{"raw":{"variants":["Pre-war models lose 27 points on invasion-era court rulings","Backward transfer holds while forward legal drift collapses","Chronological learning retains old knowledge lifts new scores","Legal variants cut temporal degradation asymmetry in judgments","Cross-epoch matrix shows additive change in Ukrainian decisions"]},"model":"grok-4.3","cost_usd":0.003467,"raw_usage":{"total_tokens":1906,"prompt_tokens":823,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":34674500,"prompt_tokens_details":{"text_tokens":823,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1012,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":823,"tokens_out":71,"duration_ms":10667,"temperature":1.0,"reasoning_tokens":1012,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T13:24:21.601718+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Re-running the 3x3 matrix after explicitly balancing case-type distributions, court levels, and procedural metadata across epochs and finding that the forward degradation disappears would falsify the claim that temporal language evolution is the cause.","supporting_citations":[],"review_version":1}