{"id":"7a2ae17f-b5e6-4c86-a4b9-4c62cb4be91a","arxiv_id":"2505.02983","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A logits-constrained decoding mechanism on top of GujiRoBERTa improves ancient Chinese NER F1 by up to 2.95% on two of three EvaHan 2025 datasets, but the model-selection scaling law is fitted to four datasets and lacks validation.","lead":"This paper proposes a logits-masking decoding method that enforces valid label transitions for ancient Chinese named entity recognition. It reports modest F1 gains over CRF baselines on two of three shared-task datasets, and presents a fitted rule for choosing models based on label count and data size.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LC advantage in high-label NER rests on single-run F1 gaps of 0.5–2.0 points with no variance estimate; the stated 1.95% average does not reproduce from Tables 2–4.","rationale":"The reader's weakest assumption is exactly that the experimental ranking is stable despite single runs without error bars. I agree, and I verified the concern against the paper's own tables. The stated 1.95% average improvement cannot be recovered from Tables 2 and 4 under either absolute or relative calculations. Moreover, the tables themselves are internally inconsistent with the prose: on Dataset C, CRF+LC outperforms LC-only, and on Hybrid B, CRF alone outperforms CRF+LC, contradicting the §4.5 claim that the combination surpasses individual models. These issues do not disprove the underlying idea—logits-constrained decoding is a plausible, parameter-free way to enforce BMES constraints, and the Limitations section honestly notes sensitivity to the initial token—but they mean the central empirical claim is not yet established. A multi-seed rerun with confidence intervals is the minimal check that would settle whether the reported gaps are signal or noise. Since this is the same weakness the reader identified, and the reader's CONDITIONAL verdict appropriately asks for that evidence before accepting the claim, I see no reason to change the verdict.","tokens_in":4989,"tokens_out":7830,"duration_ms":76873,"concrete_test":"Run each configuration in Tables 2–4 and Table 6 with 5 different random seeds, keeping all hyperparameters from §4.2 fixed, and report mean and standard deviation of F1 for every configuration. Then compute paired bootstrap 95% confidence intervals for LC−plain and LC−CRF on Datasets A and C, and recompute the claimed average improvement. If the interval for Dataset A includes 0, or if the reproduced average differs from 1.95 points by more than 0.3 points, the 'significant advantage' claim is not supported and the conclusion should be downgraded accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.4—that LC gives a significant advantage on L≥6 datasets with an average F1 gain of 1.95%—is supported only by Tables 2 and 4. Recomputing from those tables, Dataset A improves by 0.89 F1 points (0.9180→0.9269) and Dataset C by 1.95 points (0.8067→0.8262); the average absolute gain is (0.89+1.95)/2 = 1.42 points, not 1.95. If relative gains are used, the average is (0.97%+2.42%)/2 = 1.70%, still not 1.95%. The paper reports no number of seeds, standard deviations, or significance tests anywhere in §4, so a 0.89-point difference on a fine-tuned transformer is within typical run-to-run noise. Two further internal inconsistencies undermine the comparative story: on Dataset C, CRF+LC (0.8314) beats LC-only (0.8262), and on Hybrid B, CRF alone (0.9465) beats CRF+LC (0.9439), contradicting the §4.5 claim that the combination surpasses all individual models. Thus the only evidence for the headline advantage is a set of single-run scores whose ranking is not demonstrably stable. This is a correctness-risk issue, not a disagreement with prior work: the proposed masking itself may help, but the paper does not currently establish that it does.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Logits-Constrained (LC) decoding framework for Ancient Chinese NER. The model uses GujiRoBERTa to produce token representations and a linear classifier, then applies a hand-built BMES transition mask to the logits at inference time, avoiding CRF and BiLSTM components. Experiments are reported on the EvaHan 2025 datasets A, B, C and an expanded Hybrid B, with eight configurations per dataset. The paper claims that LC outperforms CRF-based methods in high-label scenarios, that BiLSTM integration is harmful, and that a label-count/data-size criterion of the form 0.16L^2.8 can guide model selection.","tokens_in":5289,"tokens_out":6864,"duration_ms":70728,"significance":"If the empirical claims held, this would be a useful and refreshingly simple contribution: the LC mask adds no trainable parameters, enforces structural constraints directly, and is easy to reproduce from the description. The ablation design covers the standard NER baselines, the hybrid-data expansion is a sensible robustness check, and the limitations section is candid about the initial-token sensitivity and segmentation dependence. However, the present evidence is not strong enough to support the central claims. All comparisons are single-run F1 scores without variance estimates, the headline 1.95% average improvement cannot be recovered from Tables 2 and 4, and the claim that CRF+LC is optimal is contradicted by Table 6. The model-selection criterion in Section 4.6 is fit in-sample on the same four datasets used for demonstration. The proposed mechanism is plausible, but the paper as written does not establish that it helps.","major_comments":[{"comment":"The headline claim that LC achieves an average F1 improvement of 1.95% on L≥6 datasets does not reproduce from the reported numbers. In Table 2, Dataset A improves from 0.9180 to 0.9269 (+0.0089 absolute); in Table 4, Dataset C improves from 0.8067 to 0.8262 (+0.0195 absolute). The average absolute gain is 1.42 F1 points, and the average relative gain is 1.70%. Additionally, the text states that Dataset C rises by '+2.95%', whereas the relative gain is 2.42% and the absolute gain is 1.95 points; the abstract and conclusion repeat the 2.95% figure. The abstract, §4.4, and §5 therefore rest on a number that does not match the supporting tables.","section":"§4.4, Tables 2 and 4"},{"comment":"All results are from single runs; no seeds, standard deviations, confidence intervals, or significance tests are reported anywhere in Section 4. The Dataset A LC-versus-baseline gap is only 0.89 F1 points, which is within typical run-to-run variation for fine-tuned transformer models. Without repeated runs or statistical testing, the statement that LC 'exhibits a significant advantage' is not supported, and the same applies to the claims that BiLSTM 'leads to performance degradation' with an average ΔF1=−3.8%. The later '82% of performance variance' statement is not a substitute because it describes an in-sample fit rather than experimental uncertainty.","section":"§4.4, Tables 2–4"},{"comment":"The text states that the combined CRF+LC framework 'surpasses the performance of individual framework implementations,' but Table 6 reports CRF-only F1 of 0.9465 versus CRF+LC of 0.9439 on Hybrid B. The same pattern occurs on Dataset A, where LC-only gives 0.9269 and CRF+LC gives 0.9213. Thus the claim that CRF+LC is the optimal configuration is contradicted by the paper's own tables, and the abstract's general claim about 'large-data settings' is not supported, since LC-only underperforms CRF-only on the expanded Hybrid B.","section":"§4.5, Table 6"},{"comment":"The model-selection criterion is circular as evidence. The parameters α=0.16 and β=2.8 are optimized via Eq. (4) against the F1 values of exactly the four datasets (A, B, C, Hybrid B) on which the criterion is then demonstrated, and the '82% of performance variance' statement describes the fit of that same in-sample regression. No held-out dataset or cross-validation is provided, so the claimed predictive power of the rule is not established. In addition, Eq. (5) is presented without any fitting procedure or goodness-of-fit information.","section":"§4.6, Eq. (3)–(5)"}],"minor_comments":[{"comment":"The notation 'S3 = Perm({B,M,E})' is undefined and confusing; the claim that '(B,M,E) is the only valid transition in S3' is inaccurate under the BMES scheme, since legal sequences can also contain S-tagged tokens and O labels.","section":"§2.2"},{"comment":"There are typographical errors in 'maps tok possible labels' and 'senarios' in §2.2; these should be corrected.","section":"§3.2.1"},{"comment":"Equation (2) is applied at inference only, since training uses standard cross-entropy without transition modeling; the term 'differentiable masking' is therefore misleading unless the refinement is also used during training, which the text does not state.","section":"§3.2.2"},{"comment":"Dataset statistics are incomplete: Table 5 gives sentence counts only for Dataset B and Hybrid B, not for Datasets A and C, although the model-selection rule depends on sentence count N. The paper should also state the evaluation split and any preprocessing details needed for reproducibility.","section":"§4, Tables 1–6"},{"comment":"The sentence 'the performance of CRF decreases by 1.3% and 0.5% on Datasets A and C' does not match Tables 2 and 4: CRF-only versus baseline changes are −0.40% and +0.56% in relative terms, respectively; the intended comparison should be clarified.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially an EvaHan system report, and the core method is simple. The load-bearing issues are empirical validation and internal consistency, not the architecture itself. I would encourage the editor to require multi-seed experiments with variance reporting, a corrected recalculation of the average improvement, and out-of-sample validation (or at least cross-validation) of the model-selection rule before considering the paper further. If the authors cannot provide these, the manuscript may be better suited to a shared-task report format."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe central claim—logits masking beats CRF in high-label ancient Chinese NER—is plausible but not proven by this paper. The strongest piece of evidence, actually, is the negative result: BiLSTM consistently hurts their GujiRoBERTa setup, by roughly 2–4 F1 points on every dataset. That was worth measuring.\n\nWhat's new is narrow: known logits-masking (Jiang et al. 2021, Wei et al. 2021, both cited) applied to EvaHan 2025 ancient Chinese NER, plus a heuristic model-selection rule. The rule is a two-parameter fit to the exact datasets used for evaluation, with no held-out validation. The '82% of performance variance' is in-sample, so I would treat the scaling law as descriptive, not predictive.\n\nThe soft spots are real. There are no seeds, standard deviations, or significance tests anywhere in Section 4. The average improvement on L≥6 datasets is 1.42 F1 points by my arithmetic from Tables 2 and 4, not the 1.95% the text claims. And Section 4.5 says CRF+LC 'surpasses individual framework implementations,' but Table 6 shows CRF alone beats CRF+LC on the hybrid B dataset (0.9465 vs 0.9439). These are overstatements, not necessarily disqualifying: averaged over all four datasets, CRF+LC does edge out CRF alone. But the specific numbers should be fixed.\n\nThe authors do state sensible limitations (first-token sensitivity, inference overhead, segmentation dependence), which is more honest than most competition reports. The citation pattern is fine; they cite the masking works they are extending.\n\nThis is a workshop-level contribution, useful to people working on ancient Chinese NER or on constrained decoding with PLMs. It does not change the field, but it is a legitimate data point. For peer review: send it out, but with a clear request for major revision. Multiple seeds, error bars, and a corrected summary are necessary before the central claim is accepted.","headline":"Plausible but unproven case for logits masking in ancient Chinese NER, with a useful BiLSTM negative result and numbers that don't match its tables.","tokens_in":5864,"tokens_out":5107,"would_cite":false,"duration_ms":50591,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that masking illegal BMES transitions at decoding time, with no added parameters, beats conditional random fields on ancient Chinese NER when datasets have many entity categories.","keywords":["ancient Chinese NER","logits-constrained decoding","BMES tagging scheme","conditional random field","GujiRoBERTa","sequence labeling","model selection","EvaHan 2025"],"falsifier":"Run the plain, CRF, LC, and CRF+LC configurations on Datasets A and C ten times each with different random seeds and report mean F1 with standard deviations; if the mean LC advantage over CRF on the six-category datasets is smaller than the configuration-to-configuration standard deviation, the central high-label claim is not supported.","tokens_in":4745,"feed_emoji":"📜","tokens_out":8677,"duration_ms":85660,"temperature":0.7,"pith_summary":"This paper claims that a simple, parameter-free constraint—masking out every label transition that the BMES tagging scheme (Begin/Middle/End/Single entity positions) forbids—is an effective decoding strategy for ancient Chinese named entity recognition, and that it beats conventional conditional random fields exactly in the settings where CRFs tend to fail: datasets with several entity categories. On the EvaHan 2025 data, the logits-constrained (LC) model raises F1 by an average of 1.95 percentage points on the two six-category datasets, with the largest gain on Dataset C (0.8067 to 0.8262). The paper also reports that adding a BiLSTM to the pretrained encoder consistently lowers F1, and it derives a practical selection rule based on label count and sentence count. A sympathetic reader would care because the improvement comes from a hand-specified rule inserted at zero learnable cost, not from a larger model.","feed_headline":"For six-category ancient Chinese NER, a logits mask beats CRF","feed_subtitle":"Masking illegal BMES transitions adds no parameters yet lifts F1 by 1.95 points on average in high-label runs.","key_machinery":"The load-bearing object is the constraint matrix $M\\in\\{0,1\\}^{k\\times k}$, a hand-built 0/1 table over the $k$ label classes in the BMES scheme (B=Begin, M=Middle, E=End, S=Single). At each decoding step the previous predicted label selects row $M[y_{t-1}]$, and the next logits are refined by $l'_t = M[y_{t-1}] \\odot l_t + (1-M[y_{t-1}])\\cdot(-\\infty)$, sending illegal transitions to negative infinity. The mask is not learned and adds no parameters; it sits between a GujiRoBERTa encoder plus linear projection and the hard label choice, and it is what lets the pipeline enforce structural validity without CRF training.","core_discovery":"The paper's central claim is that dynamic masking of next-token logits by a fixed BMES-valid transition matrix $M$ is a viable substitute for the learned transition parameters of a CRF. The authors argue this is especially true when label complexity is high: with six NER categories on Datasets A and C, the LC configuration reaches an average F1 gain of 1.95% over non-LC configurations, and on Dataset C the gain over the plain GujiRoBERTa baseline is 2.95%. They further claim that the combined CRF+LC model is best in moderate settings, beating both components alone on Dataset B and on the expanded hybrid dataset, and that a rule of thumb $\\Gamma(L,N)$—use LC alone only when $L\\ge20$ and $N>0.16L^{2.8}$, otherwise CRF+LC—summarizes the experiments.","pith_inferences":["The same differentiable-mask trick should transfer to other structured prediction tasks with known grammar, such as part-of-speech tagging or syntactic chunking, where a hand-built constraint table can replace learned transition machinery at no parameter cost.","The threshold $0.16L^{2.8}$ invites testing as a scaling law: independent ancient Chinese datasets with label counts between 13 and 25 could show whether the exponent is stable or an artifact of the four fitted points.","A soft, learnable version of $M$, suggested by the paper's own limitations, would let the constraint adapt to irregular or unpunctuated historical texts rather than relying on manual BMES rules."],"forward_implications":["On datasets with six entity categories, the logits-constrained decoding improves average F1 by 1.95 percentage points over otherwise identical models, with a 2.95-point gain on the hardest dataset.","Adding a BiLSTM to the pretrained encoder costs about 3.8 F1 points on average, so recurrent modules should be dropped from this pipeline.","For modest label counts and limited data, CRF+LC outperforms both LC alone and CRF alone (for example, F1 of 0.9308 on Dataset B), supporting the proposed regime split.","The empirical rule $\\Gamma(L,N)$ predicts that an LC-only configuration becomes preferable at around 20 or more labels and a sentence count exceeding $0.16L^{2.8}$, a bound that grows super-linearly with label complexity."],"supporting_citations":[{"why":"Introduces the BERT pretraining approach that underlies the GujiRoBERTa encoder used in all experiments.","marker":"Devlin et al., 2019"},{"why":"Provides the robustly optimized RoBERTa pretraining recipe from which GujiRoBERTa is derived.","marker":"Liu et al., 2019"},{"why":"Defines CRFs, the main baseline the logits-constrained decoder is compared against and combined with.","marker":"Lafferty et al., 2001"},{"why":"Establishes the BiLSTM-CRF baseline that motivates the cross-comparison and whose BiLSTM component is found hurtful.","marker":"Huang et al., 2015"},{"why":"Supplies the constrained transition framework that dynamically masks invalid transitions, the closest methodological precursor.","marker":"Jiang et al., 2021"},{"why":"Develops masked transition learning for tagging schemes, another precursor of the paper's masking mechanism.","marker":"Wei et al., 2021"},{"why":"Supports the sentence-segmentation preprocessing choice that affects performance on unpunctuated ancient Chinese text.","marker":"Ge, 2022"}],"fun_headline_variants":["Logits mask beats CRF in high-label ancient Chinese NER, no extra parameters","Masking illegal BMES transitions improves ancient Chinese NER over CRF","Zero-parameter logits masking beats CRF in high-label ancient Chinese NER","For ancient Chinese NER, a logits mask outperforms CRF without added parameters","Logits-constrained decoding beats CRF in ancient Chinese NER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported F1 gaps between configurations—especially the 0.89-point gain on Dataset A and the 2.95-point gain on Dataset C—are genuine differences and not run-to-run noise, because each configuration is evaluated with a single run and no significance test.","fun_headline_variants_meta":{"raw":{"variants":["Logits mask beats CRF in high-label ancient Chinese NER, no extra parameters","Masking illegal BMES transitions improves ancient Chinese NER over CRF","Zero-parameter logits masking beats CRF in high-label ancient Chinese NER","For ancient Chinese NER, a logits mask outperforms CRF without added parameters","Logits-constrained decoding beats CRF in ancient Chinese NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001545,"raw_usage":{"total_tokens":6114,"prompt_tokens":817,"completion_tokens":5297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":5194}},"tokens_in":433,"tokens_out":5297,"duration_ms":39831,"temperature":1.0,"reasoning_tokens":5194,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:37:39.896422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the plain, CRF, LC, and CRF+LC configurations on Datasets A and C ten times each with different random seeds and report mean F1 with standard deviations; if the mean LC advantage over CRF on the six-category datasets is smaller than the configuration-to-configuration standard deviation, the central high-label claim is not supported.","supporting_citations":[{"cited_title":"Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data","cited_arxiv_id":"2106.08977","evidence_quote":"Supplies the constrained transition framework that dynamically masks invalid transitions, the closest methodological precursor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Develops masked transition learning for tagging schemes, another precursor of the paper's masking mechanism."}],"review_version":1}