{"id":"4557b60e-f6fe-4a91-ab94-76a2fd67dd50","arxiv_id":"1908.08419","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"NE-LP, a sampling strategy that adds predicted segmentation loss to normalized entropy, selects more informative sentences for active learning in Chinese word segmentation on EHRs.","lead":"This paper proposes an active learning method for Chinese word segmentation on electronic health records, combining normalized entropy with a learned loss prediction score to choose which sentences to label. The method is tested on a private hospital EHR dataset and reported to beat standard uncertainty-based sampling.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed superiority over conventional uncertainty sampling is not yet established because the baseline definitions for MTE and MTM in Eqs. (2)-(3) are not standard token-level scores; the comparison may be against strawman baselines.","rationale":"The reader's conditional verdict is appropriate, but the load-bearing weak point is not primarily the loss-prediction proxy. The strongest threat to the empirical claim is that the conventional baselines themselves are not credibly specified: Eq. (2) is not a token entropy and Eq. (3) is not a token margin, and the divergence between LC and MTE curves in Figs. 5 and 6 indicates that the implemented strategies do not match the written formulas. Without code, this makes the central comparison unverifiable and potentially unfair. The reader's requested release of code and detailed protocols would directly address this, so the conditional verdict stands. The loss-prediction proxy concern is real but secondary: if the loss predictor is inert, NE-LP collapses to normalized entropy sampling, yet the abstract's black-box claim could still hold; a beta=0 ablation would settle that. I therefore keep the verdict unchanged but emphasize that the authors must specify and justify the exact baseline implementations, and ideally release code, before the claim of consistent outperformance can be accepted.","tokens_in":13588,"tokens_out":12801,"duration_ms":139032,"concrete_test":"Re-run the active learning comparison after replacing MTE and MTM with standard token-level implementations: average per-character entropy and average per-character top-two margin, computed from the CRF marginal probabilities; also run LC in both sequence-level and token-level forms. Keep the same joint model, bigram features, and validation-based selection of alpha and beta (not test-set selection). If NE-LP does not outperform all of these standard implementations across the 10 iterations, the reported 'consistent outperformance' is an artifact of the unusual baseline definitions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that NE-LP consistently outperforms conventional uncertainty-based sampling strategies. The paper's specification of those baselines is internally inconsistent. Eq. (2) defines MTE as -sum_{i=1}^N p(y*|x) log p(y*|x), where the summand does not depend on i and p(y*|x) is the whole-sequence best-path probability; this is not a maximum token entropy score. Eq. (3) defines MTM as a difference of sequence-level path probabilities, not the per-token margin used in standard active learning for sequence labeling (e.g., Marcheggiani and Artieres, 2014, ref. [23]). Moreover, because Eq. (2) is a function only of p(y*|x), MTE as written should rank sentences nearly identically to LC as written, yet Figs. 5 and 6 show noticeably different learning curves for LC and MTE. This suggests the implemented baselines differ from the equations, and without released code or an exact protocol the reader cannot verify that NE-LP is compared against genuine mainstream uncertainty strategies rather than weakened versions. If the baselines are non-standard, the abstract's claim of consistent outperformance may hold only against strawman comparators. A related but secondary concern is that no NE-only ablation (beta=0) is reported, so the contribution of the loss-prediction term to the claimed gains is not separately demonstrated; Sec. 3.4 asserts this assumption is verified but the paper only compares the combined strategy against baselines.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NE-LP, an active learning sampling strategy for Chinese Word Segmentation (CWS) on electronic health records. The method combines a normalized entropy term, intended to measure segmentation uncertainty, with a loss prediction term from a self-attention submodel attached to a BiLSTM-CRF segmenter. The joint model is trained on labeled data, and at each active learning round the highest-scoring unlabeled sentences are selected for annotation. Experiments are conducted on cardiovascular EHRs from Shuguang Hospital. The paper reports that NE-LP consistently outperforms least confidence (LC), maximum token entropy (MTE), minimum token margin (MTM), and random sampling in F1-score, under two initial labeled set ratios and with or without bigram features.","tokens_in":13971,"tokens_out":3501,"duration_ms":37864,"significance":"If the central claim were fully supported, the paper would make a useful contribution: active learning for domain-specific CWS is a practical problem, the idea of combining uncertainty with a learned loss prediction model is plausible, and the use of real Chinese EHR data with an explicit annotation bottleneck is a strength. The paper also provides useful auxiliary comparisons, including several open-source CWS tools and multiple neural segmenters, and it clearly quantifies the gains from bigram features. However, the main comparative evidence is undermined by non-standard baseline definitions in the text, a mismatch between those definitions and the reported learning curves, and the absence of statistical repeatability measures. The work is therefore better framed as a proposal with preliminary evidence than as an established improvement over conventional uncertainty sampling.","major_comments":[{"comment":"The definitions of MTE and MTM are not standard token-level uncertainty scores. In Eq. (2), the summand does not depend on i, so S_MTE(x) reduces to -N * p(y*|x) log p(y*|x), which is a function of only the best-path probability. In Eq. (3), the margin is computed between sequence-level path probabilities rather than between the top two tag probabilities at each token position, which is the conventional margin for sequence labeling as used in Marcheggiani and Artieres (2014), reference [23]. Because -N p log p is strictly decreasing in p for p in (0,1), Eq. (2) should rank sentences nearly identically to LC in Eq. (1). The learning curves in Figs. 5 and 6 show noticeably different behavior for LC and MTE, which indicates that the implemented baselines are not the ones defined in Eqs. (1)-(3). Without the code or an exact protocol, the reader cannot verify that NE-LP is compared against genuine mainstream uncertainty strategies rather than weakened versions. This directly affects the abstract's claim of consistent outperformance and should be fixed by correcting the baseline definitions and re-running, or by clearly describing the actual implementations used.","section":"Section 2.2, Eqs. (2)-(3)"},{"comment":"The weighting coefficients alpha and beta are selected by comparing learning curves in Fig. 4 on the same test set used in the main comparisons, and the paper then fixes alpha = beta = 1 for all subsequent experiments. This is a form of test-set selection that can inflate the reported gains of NE-LP over the baselines, because the hyperparameters are chosen to maximize the displayed F1 advantage. A held-out validation set or nested selection procedure should be used, and the sensitivity of the main comparison to alpha and beta should be reported. Without this, the reported margins in Figs. 5 and 6 are not a fair estimate of expected performance.","section":"Section 4.3.4, Eq. (16)"},{"comment":"The central experimental evidence consists of single learning curves with no error bars, no multiple runs, and no significance tests. The reported gaps between NE-LP and MTE are small in several iterations, especially in the 1:9 setting of Fig. 6, where the paper itself notes that the curves are close. A single run cannot support the claim that NE-LP 'always' or 'consistently' outperforms MTE. The paper should report means and standard deviations over multiple random seeds, and where appropriate paired significance tests. Additionally, no NE-only ablation (beta = 0) is reported, so the specific contribution of the loss prediction term to the claimed gains is not demonstrated. Section 3.4 states that the informativeness assumption is verified in the experiments, but the experiments only compare the combined strategy against baselines, not NE-LP against NE alone.","section":"Section 4.3.5, Figs. 5-6"},{"comment":"The sampling score in Eq. (15) is called normalized entropy, but the notation is ambiguous: pSeg(x) appears inside a sum over N classes without a token index, and the denominator contains log(1/N) * sqrt(Len), where Len is the sequence length. If pSeg(x) is the whole-sequence probability of the best tag sequence, then Eq. (15) is not a token-level normalized entropy and the denominator's dependence on sequence length is unmotivated. If pSeg(x) is instead the marginal probability of a tag at a token, then the formula needs a sum over tokens and a clear definition of how token entropies are aggregated into a sentence score. The authors should clarify the exact quantity used to rank sentences, since this is the core of the proposed method.","section":"Section 3.4, Eq. (15)"}],"minor_comments":[{"comment":"The abstract contains a typo: 'Specically' should be 'Specifically'.","section":"Abstract"},{"comment":"Line 5 reads 'label U by fτ', which is unclear for an active learning setup where U is the unlabeled pool; it should say 'predict labels for U with fτ' or 'obtain current predictions on U'.","section":"Algorithm 1"},{"comment":"The prose says MTM 'subtracts the highest probability by the lowest one'; this should read 'subtracts the second-highest probability from the highest one' to match standard margin sampling and the notation max'.","section":"Section 2.2, Eq. (3)"},{"comment":"Reference [31] contains a corrupted author field ('Kaiser, /suppress L.'); this should be corrected to the standard Vaswani et al. citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of Neural Computing and Applications, but the main contribution is an empirical heuristic and the evidence is currently not sufficient. The baseline definition mismatch in Eqs. (2)-(3) versus the reported curves is my primary concern; if the authors cannot supply code or a precise protocol showing that LC, MTE, and MTM are implemented in the conventional way, the central comparison may not be reproducible. The hyperparameter selection on the test set and the absence of variance estimates are secondary but important. I would encourage the editors to ask for a revised version with corrected baseline definitions, additional NE-only and sensitivity experiments, and at least a few random seeds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: this is a sensible engineering paper with a real problem behind it, but the evidence as presented doesn't yet support the 'consistently outperforms' claim. The core idea — combine normalized entropy with a learned loss predictor, trained jointly with a BiLSTM-CRF segmenter — is a reasonable adaptation of Yoo and Kweon (2019) to Chinese word segmentation. The application to private EHR records is useful, and the comparisons of open-source tools and model architectures are decent background. The bigram feature analysis is actually informative.\n\nThe soft spots are in the evaluation. The definitions of the baselines in Eqs. (2) and (3) don't match the standard token-level maximum token entropy and minimum token margin used in sequence labeling. Both are written as functions of the whole-sequence best-path probability p(y*|x). As written, MTE is essentially a scaled version of LC, yet Figs. 5-6 show MTE and LC behaving quite differently, which suggests the implemented baselines differ from the equations. That's a genuine reproducibility problem. You can't claim to beat mainstream uncertainty sampling unless you've compared against its actual token-level forms.\n\nThere's also no ablation that isolates the loss-prediction term: no NE-only (beta=0) or LP-only (alpha=0) curve. The weight-coefficient experiment varies alpha and beta but never sets one to zero, so the contribution of the new component isn't demonstrated. The alpha/beta selection also seems to be done on the data used for the main comparison, which can inflate gains. And the paper reports single runs with no variance or significance testing, and no code/data release. These are fixable, but they need to be addressed.\n\nFor who: this is for researchers working on active learning for clinical Chinese NLP. It's a plausible step, but not a conceptual one. A serious referee could force the authors to correct the baselines, run the ablations, and report variance. I'd send it to review rather than desk reject, because the problem is important and the method is reasonable.\n\nIf I were the editor, I'd ask for a revised version with exact baseline definitions, NE-only and LP-only curves, and confidence intervals.","headline":"A reasonable active-learning recipe for clinical Chinese word segmentation, but the claimed gains over uncertainty baselines aren't established because the baselines look mis-specified and the key ablation is missing.","tokens_in":14405,"tokens_out":4228,"would_cite":false,"duration_ms":40470,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Active learning that scores sentences by normalized entropy plus a predicted segmentation loss selects more informative electronic health record sentences than standard uncertainty sampling, as measured by F1 over active-learning rounds.","keywords":["active learning","Chinese word segmentation","electronic health records","loss prediction","normalized entropy","BiLSTM-CRF","self-attention","bigram features"],"falsifier":"Fix the segmenter and replace the loss-prediction term with a control trained on randomly shuffled segmentation losses; if NE-LP's F1 advantage over normalized-entropy-only sampling does not disappear or shrink sharply, then the reported gain is not caused by loss prediction. A direct check is to plot, on held-out sentences, the correlation between predicted loss and the segmenter's actual loss after retraining, and see whether the top-ranked sentences by NE-LP actually produce the largest F1 gains.","tokens_in":13430,"feed_emoji":"🏥","tokens_out":5755,"duration_ms":57413,"temperature":0.7,"pith_summary":"Chinese electronic health records contain no word delimiters, so clinical text needs segmentation models trained on labeled data, but medical annotation is slow because it requires expert annotators. This paper proposes an active-learning sampling strategy, NE-LP, that scores unlabeled sentences by a weighted sum of two signals: normalized token entropy from a BiLSTM-CRF segmenter and a predicted segmentation loss from a self-attention head trained jointly with the segmenter. On a cardiovascular EHR corpus, the authors report that selecting sentences by this score consistently yields higher F1 than least confidence, maximum token entropy, minimum token margin, or random selection across active-learning rounds. The intended payoff is the same segmentation quality from fewer expert-annotated sentences, which matters for specialized clinical domains where large labeled corpora are unavailable.","feed_headline":"Entropy plus predicted loss picks better EHR sentences for labeling","feed_subtitle":"A joint segmenter and loss predictor cuts the annotation burden for clinical Chinese word segmentation.","key_machinery":"The load-bearing object is a joint model: a BiLSTM-CRF word segmenter with an attached self-attention loss-prediction head. The two share the BiLSTM encoder, and the joint objective is $L_{joint}=L_{Seg}+\\lambda L_{Loss}$, where $L_{Loss}$ is the mean squared error between the predicted loss and the true segmentation loss of the segmenter. Inputs are character embeddings concatenated with pretrained bigram embeddings, which capture interactions between adjacent characters; on this corpus bigram features outperform trigram and four-gram because two-character words dominate the data. During active learning, the same joint model computes both the normalized entropy needed for the uncertainty term and the predicted loss, so no separate selection model needs to be trained. The NE-LP score in Equation (16) is the mechanism that turns raw model outputs into a ranking of sentences for annotation.","core_discovery":"The central claim is that the NE-LP score, $S_{NE-LP}(x)=\\alpha\\,Uncertainty(x)+\\beta\\,Loss(x)$, identifies more useful sentences for the next labeling round than uncertainty alone. Uncertainty is the entropy of the segmenter's per-character tag distribution over the BMES labels, normalized by $\\log(1/N)$ and scaled by $1/\\sqrt{Len}$; loss is the output of a self-attention layer trained to reproduce the segmenter's actual segmentation loss on labeled data. The authors demonstrate the claim on an EHR dataset by comparing learning curves over ten active-learning rounds at two initial-label ratios, 3:7 and 1:9. In every reported round and both ratios, NE-LP reaches a higher testing-set F1 than LC, MTE, MTM, and RAND, with the advantage over MTE larger when the initial labeled set is larger.","pith_inferences":["An untested extension: the loss predictor's marginal value may shrink as the seed set shrinks, because the reported gap between NE-LP and MTE narrows under the 1:9 split; measuring the gap across several seed sizes would show whether loss prediction is most useful once a moderate labeled base exists.","The $1/\\sqrt{Len}$ normalization is an ad hoc scaling, so a natural follow-up is to compare NE-LP with per-length normalization or learned combination weights to see whether sentence length distorts the entropy-loss sum.","Because the loss predictor is trained on labels also used to train the segmenter, its usefulness on unlabeled sentences depends on extrapolation; long sentences, rare medical terms, and out-of-vocabulary drugs would stress the predictor and could reveal whether the reported gain persists outside the original EHR distribution."],"forward_implications":["If NE-LP is correct, fewer expert-labeled EHR sentences are needed to reach a target segmentation F1, lowering annotation cost in medical NLP.","The method remains ahead of uncertainty baselines when the initial labeled set is small (1:9 ratio), suggesting it helps in especially low-resource settings.","Bigram features improve the segmenter on this corpus, and NE-LP benefits from them; the gain is explained by the prevalence of two-character words in clinical text.","The loss-prediction component contributes beyond normalized entropy, since setting both weights to 1 outperforms weighting either signal alone, and the authors state the method is intended to transfer to other sequence labeling tasks such as named entity recognition and relation extraction."],"supporting_citations":[{"why":"Supplies the learning-loss-for-active-learning training paradigm that the self-attention loss prediction head is built on.","marker":"[38]"},{"why":"Provides the self-attention mechanism used in the loss prediction submodel and in the Transformer-CRF comparison model.","marker":"[31]"},{"why":"Justifies choosing BiLSTM as the base segmentation architecture, which the joint model shares with the loss predictor.","marker":"[22]"},{"why":"Establishes the existing active-learning line for Chinese word segmentation that this paper extends to the medical domain.","marker":"[19]"},{"why":"Defines uncertainty sampling and the least-confidence baseline against which NE-LP is compared.","marker":"[18]"},{"why":"Defines the minimum-token-margin sampling strategy used as one of the compared baselines.","marker":"[23]"}],"fun_headline_variants":["Entropy plus loss prediction picks best EHR sentences for labeling","NE-LP sampling cuts annotation cost for clinical word segmentation","Active learning for EHR CWS: normalize entropy, predict loss, win","Hybrid sampling outperforms uncertainty for Chinese EHR segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on the assumption that a loss predictor trained on the current labeled set can judge which unlabeled sentences will be hardest for the segmenter; if predicted loss is a poor proxy on unlabeled data, NE-LP degenerates to normalized entropy sampling.","fun_headline_variants_meta":{"raw":{"variants":["Entropy plus loss prediction picks best EHR sentences for labeling","NE-LP sampling cuts annotation cost for clinical word segmentation","Active learning for EHR CWS: normalize entropy, predict loss, win","Hybrid sampling outperforms uncertainty for Chinese EHR segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1387,"prompt_tokens":952,"completion_tokens":435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":568,"tokens_out":435,"duration_ms":4623,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:39:10.590805+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix the segmenter and replace the loss-prediction term with a control trained on randomly shuffled segmentation losses; if NE-LP's F1 advantage over normalized-entropy-only sampling does not disappear or shrink sharply, then the reported gain is not caused by loss prediction. A direct check is to plot, on held-out sentences, the correlation between predicted loss and the segmenter's actual loss after retraining, and see whether the top-ranked sentences by NE-LP actually produce the largest F1 gains.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the learning-loss-for-active-learning training paradigm that the self-attention loss prediction head is built on."},{"cited_title":"In: Advances in Neural Information Processing Systems, pp","cited_arxiv_id":null,"evidence_quote":"Provides the self-attention mechanism used in the loss prediction submodel and in the Transformer-CRF comparison model."},{"cited_title":"In: Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pp","cited_arxiv_id":null,"evidence_quote":"Justifies choosing BiLSTM as the base segmentation architecture, which the joint model shares with the loss predictor."},{"cited_title":"In: Proceedings of International Conference on Computational Linguistics 2012: Posters, pp","cited_arxiv_id":null,"evidence_quote":"Establishes the existing active-learning line for Chinese word segmentation that this paper extends to the medical domain."},{"cited_title":"In: Proceedings of the 17th An- nual International Conference on Research and Develop- ment in Information Retrieval, pp","cited_arxiv_id":null,"evidence_quote":"Defines uncertainty sampling and the least-confidence baseline against which NE-LP is compared."},{"cited_title":"In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp","cited_arxiv_id":null,"evidence_quote":"Defines the minimum-token-margin sampling strategy used as one of the compared baselines."}],"review_version":1}