{"id":"8cccecdb-9861-48ba-a253-01ad81788f64","arxiv_id":"2411.11623","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new federated incremental NER setup is introduced, along with a LGFD model using structural distillation, contrastive loss, and an entropy-based task switch detector that improves average F1 over existing INER baselines.","lead":"This paper defines a new setup, Federated Incremental NER, where entity types and local clients change over time, and proposes a model that counters forgetting through structure distillation, contrastive learning, and a task switching monitor. The work matters because it extends privacy-preserving federated learning to dynamic, evolving NER tasks, with reported gains over existing incremental NER baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The task switching monitor is the least-supported load-bearing component: detection accuracy and threshold sensitivity are untested, and Algorithm 1 contradicts Section IV-C on which model to store as teacher.","rationale":"The Reader's verdict identifies the entropy-jump threshold in the task switching monitor as the weakest assumption, and my stress-test lands on the same component. I sharpened it with two manuscript-internal observations: (a) Section IV-C and Algorithm 1 disagree about whether the stored teacher is Theta^{r-1,t} or Theta^{r,t}, so even reproduction is ambiguous; (b) no direct evaluation of detection quality is reported, so the effect of threshold failures on LSKD and LITC is unquantified. This concern is load-bearing because the monitor is the only mechanism that selects the old global model, and that model feeds both the structural distillation loss and the pseudo-label-guided contrastive loss. However, the concern does not warrant changing the verdict: the paper's ablations do show the losses matter, and the aggregate F1 improvements over strong baselines are consistent across settings, so the central claim remains plausible pending the missing detection evidence. A conditional verdict with a request for detector diagnostics and the oracle comparison is the appropriate outcome. The released code makes the proposed test feasible, and I would not move to accept or reject on the current evidence.","tokens_in":23816,"tokens_out":18054,"duration_ms":174992,"concrete_test":"Reproduce the OntoNotes5 10-1 setting with ground-truth switch rounds logged: (1) record the monitor's per-round decisions and compute switch-detection F1, missed-detection rate, and mean delay; (2) compare monitor-based LGFD against an oracle variant that stores the true previous-task model at the real boundary using the same code path; (3) sweep lambda over {0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2} and rerun the setting. If the oracle variant beats the monitor variant, or if detection F1 is low, the headline gains are not attributable to the automatic detection mechanism; if performance collapses for any lambda near 0.6, the fixed threshold is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The inter-client forgetting defense and the old-teacher supply for both LSKD and LITC depend entirely on the task switching monitor (Section IV-C, Equation 8). Its correctness assumes that a new entity type always produces an entropy jump of at least lambda=0.6 on the client's local data. The paper never reports monitor precision/recall, detection delay, or a lambda sensitivity study, and no experiment uses genuinely irregular arrival schedules. This matters because Equation (7) and the pseudo-labeling in Equation (3) use the stored old model as teacher; a missed or false switch stores a polluted/wrong teacher, degrading every component that Table VI credits for the gains. The manuscript is also internally inconsistent: Section IV-C says to store global model Theta^{r-1,t} from the previous round, while Algorithm 1 stores the just-received Theta^{r,t}. The main-results comparison therefore cannot separate LGFD's losses from the monitor's behavior on the specific alphabetical schedules, leaving the claimed automatic privacy-preserving detection as the least-supported load-bearing element.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines Federated Incremental NER (FINER), a setting in which local clients receive new entity types incrementally and new clients may join the federated training process, and proposes a Local-Global Forgetting Defense (LGFD) model. The method combines a structural knowledge distillation loss (LSKD), a pseudo-label-guided inter-type contrastive loss (LITC), and a task switching monitor that detects new entity types from average-entropy jumps and stores a global old model as a teacher for distillation and pseudo-labeling. Experiments on I2B2 and OntoNotes5 under four settings (8-1, 8-2, 10-1, 10-2) report consistent improvements in average Mi-F1 and Ma-F1 over INER baselines, with ablations showing both LSKD and LITC contribute, and a robustness experiment over 10 random entity-type orders.","tokens_in":23944,"tokens_out":5981,"duration_ms":63150,"significance":"If the central claim holds, the paper introduces a practical and timely benchmark and a method that outperforms prior INER approaches under a federated, privacy-preserving incremental regime. The manuscript has notable strengths: the code is released, the experiments cover multiple settings and datasets, ablations isolate the main loss components, and the random-order analysis addresses an important robustness concern. However, the empirical claim is conditional on the task switching monitor, which is load-bearing but never directly validated, and on resolving an internal inconsistency about which checkpoint is stored as the old teacher. The significance of the contribution is therefore real but currently under-supported by the evidence presented.","major_comments":[{"comment":"Section IV-C states that the monitor should store the latest global model Theta^{r-1,t} from the (r-1)-th global round as the old model Theta^{t-1}, while Algorithm 1 explicitly stores the just-received Theta^{r,t} as the old model. These are different checkpoints, and the stored model is the teacher for LSKD, LITC, and pseudo-labeling. The manuscript must specify which checkpoint is actually used, correct the algorithm or text accordingly, and confirm that the reported results correspond to the corrected implementation.","section":"Section IV-C vs. Algorithm 1"},{"comment":"The task switching monitor is never validated as a detector. The paper reports no precision, recall, detection delay, or threshold sensitivity for the entropy-jump criterion I^{r,t}_m - I^{r-1,t}_m >= lambda with lambda=0.6, even though misses and false alarms directly corrupt the stored old model used by LSKD, LITC, and pseudo-labeling. Moreover, the experiments in Section V.B use alphabetical entity orders and add exactly 4 new clients per task, so neither irregular client arrival nor irregular switch timing is exercised. Please add (i) monitor precision/recall against oracle switch times, (ii) a lambda sweep, (iii) an ablation that bypasses the monitor with oracle old-model selection, and (iv) at least one setting with irregular arrival schedules.","section":"Section IV-C, Eq. (8)"},{"comment":"The structural knowledge distillation loss compares left singular vectors U from SVD of old and current hidden states via MSE. SVD singular vectors are determined only up to sign (and up to rotation for repeated singular values), so MSE(U^{t-1}, U^t) is not a well-defined subspace distance unless signs are aligned. No sign alignment or projection-based formulation is described. Since the ablation in Table VI shows that replacing LSKD with LFD hurts performance, this issue is load-bearing; the authors should address it, for example by comparing projection matrices UU^T or by using a sign-invariant distance.","section":"Eq. (2), LSKD"},{"comment":"The paper reports that each experiment is run 3 times with 3 seeds, but Tables II-V contain only averaged numbers with no standard deviations, confidence intervals, or per-seed results. The paired t-test at a 0.05 significance level is therefore based on n=3, and the significance markers (†, ‡, ♮) cannot be properly assessed without variance information. Figure 4 shows boxplots over 10 entity-type orders but does not clarify whether the boxplots aggregate over seeds or orders. Please provide per-seed results or error bars and a precise description of how the paired t-test was computed.","section":"Section V.C and V.E"}],"minor_comments":[{"comment":"The role of newly joined clients S_n in Equation (7) is unclear: the text includes S_n in the set of clients using the old global model for LSKD and LITC, but new clients have no prior local tasks and may not have a meaningful old model Theta^{t-1}. Please clarify what the old model is for these clients.","section":"Section IV-A"},{"comment":"The confidence threshold alpha_e is mentioned in the text but never defined precisely; please specify how the median entropy is computed and over which set of tokens or entity types.","section":"Eq. (3)"},{"comment":"The sensitivity analysis covers only lambda_1. Given that the monitor threshold lambda is a load-bearing hyperparameter, please add sensitivity experiments for lambda and preferably also for lambda_2 and G.","section":"Table VII"},{"comment":"The case Task=False is not explicit in Algorithm 1; please clarify whether the local update then uses Equation (7) without the old-model losses or uses a different objective.","section":"Algorithm 1"},{"comment":"The heading 'Language Language Models' appears to be a typo and should read 'Large Language Models'.","section":"Section II.C heading"},{"comment":"The caption of Figure 4 should state whether the boxplots are computed over the 10 random orders, over the 3 seeds, or over both, and should define the boxes and whiskers.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"I would additionally ask the editor to give extra scrutiny to the comparison with CPFD+FL, since CPFD is the authors' own prior work and the federated adaptation of the baselines is not described in detail. This is a standard fairness concern rather than a reason for rejection on its own. The paper might also be strengthened by including an adapted federated class-incremental baseline from the related work cited as [31]-[33], since those methods address a closely related problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, read this one if you care about continual NER meeting federated learning. The paper's real contribution is the FINER setup: incremental NER under federated constraints with clients arriving mid-process and no one knowing when a new entity type appears. That setup is new, and the LGFD model beats the strongest INER baseline (CPFD, the authors' own prior work) by 2-9 F1 points across four settings on I2B2 and OntoNotes5. The gains hold up in ablations: removing either loss or swapping SKD for vanilla feature distillation drops performance. They also tested 10 random entity orders and the advantage over CPFD+FL is stable. Code is public. That is a solid package.\n\nThe soft spots are real but localized. The task switching monitor is the load-bearing piece for the automatic privacy-preserving detection claim, and it is the least tested piece. We never get precision/recall or detection delay for the monitor, no sensitivity analysis for the entropy threshold lambda=0.6, and all experiments use fixed alphabetical 1-1 or 2-2 schedules, so 'irregular client arrival' is not actually exercised. There is also an internal contradiction: Section IV-C says to store the previous-round global model Theta^{r-1,t} as the old teacher, while Algorithm 1 stores the just-received Theta^{r,t}. That is exactly the model that feeds both distillation and pseudo-labeling, so the paper needs to fix this and show the monitor actually detects switches as intended.\n\nLess seriously, the three runs are averaged without error bars or per-seed numbers, and the significance tests are only against CPFD+FL for the headline averages. The hyperparameters are fixed defaults, which is fine, but lambda1 sensitivity is only shown on one setting.\n\nNone of these sink the paper. The intra-client losses earn their keep in the ablations, and even a skeptical reader would see the improvement pattern as real. But the inter-client component, the part that makes this 'federated' rather than just 'incremental with extra clients', needs direct validation before the claims fully stand.\n\nRecommendation: this deserves a serious referee. The setup alone will be cited; the authors should be pushed to resolve the Algorithm 1/IV-C conflict and to add a monitor diagnostic (detection accuracy, delay, lambda sweep) plus error bars. If they do that, this is a solid conference paper. Bring it to reading group if your group works on continual learning or federated NLP.","headline":"FINER setup is a genuine new problem and LGFD shows real gains; the task switching monitor is the load-bearing but least-tested piece, and the paper contradicts itself on the stored teacher.","tokens_in":24541,"tokens_out":3067,"would_cite":true,"duration_ms":29969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new training scheme lets federated named-entity-recognition systems keep learning new entity types without forgetting old ones, all without sharing private data.","keywords":["federated learning","incremental named entity recognition","catastrophic forgetting","knowledge distillation","contrastive learning","pseudo-labeling","task switching monitor","entity type drift"],"falsifier":"Run a FINER experiment where a new entity type is introduced whose tokens the old model already predicts with low entropy, so the averaged entropy gap never reaches 0.6 while forgetting still occurs; if task switches are missed and final F1 degrades, the monitor assumption is falsified. Alternatively, inject noisy rounds into the local data and check whether false entropy jumps above 0.6 store an outdated teacher and reduce final performance.","tokens_in":23549,"feed_emoji":"🧠","tokens_out":2787,"duration_ms":29111,"temperature":0.7,"pith_summary":"This paper defines Federated Incremental NER (FINER), a realistic setting where local clients continuously receive new entity types while new clients join, old training data is unavailable, and data distributions differ across clients. It claims that forgetting in this setting can be countered by a Local-Global Forgetting Defense (LGFD) model, built from three components: structural knowledge distillation over SVD-projected feature subspaces, pseudo-label-guided inter-type contrastive learning, and a task switching monitor that detects new entity types from local prediction entropy. On the I2B2 and OntoNotes5 datasets, across four FINER settings, LGFD outperforms prior incremental NER methods combined with federated learning, with gains up to 8.99 Micro-F1 points over the strongest baseline. If the claim holds, privacy-preserving NER systems can keep adapting to new entity types over time without retraining from scratch or exposing raw data.","feed_headline":"New model lets federated NER learn new entity types without forgetting","feed_subtitle":"LGFD blocks old-entity forgetting across clients with SVD distillation, pseudo-label contrast, and an entropy-based task switch.","key_machinery":"The mechanism is a three-loss objective plus a privacy-safe monitor. A structural knowledge distillation loss $\\mathcal{L}_{\\mathrm{SKD}}$ applies SVD to hidden-state partitions and matches the resulting subspace structures between old and new models. A pseudo-label-guided inter-type contrastive loss $\\mathcal{L}_{\\mathrm{ITC}}$ compares entity-type prototypes computed from old and new hidden states, with pseudo-labels supplied by a confidence-based strategy from the old model. The task switching monitor computes the averaged prediction entropy $I_{m}^{r,t}$ on a client's local data each round and declares a task switch when $I_{m}^{r,t} - I_{m}^{r-1,t} \\ge 0.6$, at which point the latest global model is stored as the teacher for distillation and pseudo-labeling.","core_discovery":"The central claim is that heterogeneous forgetting in FINER—old entity types fading inside each client and across clients—can be mitigated without the server knowing when or which clients receive new entity types. The model achieves this by distilling the geometric structure of hidden representations rather than raw features, by using confident pseudo-labels from the old model to re-label old entity types hidden inside the current non-entity class and then contrasting their prototypes, and by monitoring a local entropy jump to decide when a task switch occurred, storing the last global model as the teacher. Across four settings, this design yields the new state of the art in final average Micro-F1 and Macro-F1 scores compared with prior INER methods combined with federated learning.","pith_inferences":["The fixed entropy threshold of 0.6 is likely to need adaptation across domains and model scales, since prediction entropy scales with label-set size and confidence calibration.","A testable extension is to replace the hard threshold with a client-specific or adaptive detector, using e.g. relative entropy growth or change-point detection on the monitored signal.","Because the monitor's stored old model drives both distillation and pseudo-labeling, a single false switch or missed switch should degrade all downstream gains; measuring this sensitivity is a natural stress test.","The method's reliance on confident pseudo-labels suggests it may benefit from calibration or rejection rules when old-model predictions are systematically overconfident on new entity types."],"forward_implications":["The FINER setup provides a shared benchmark for future work on dynamically evolving federated NER, including new clients joining irregularly.","SVD-based structural distillation is shown, by ablation, to preserve old knowledge better than vanilla feature distillation.","Pseudo-label-guided prototype contrast improves discrimination among entity types after the non-entity class has absorbed old and future types.","LGFD remains stable under random permutations of entity type order, not just the alphabetical order used in the main experiments.","The full defense requires only per-client entropy statistics on the server, so no private ground-truth labels or task schedules are communicated."],"supporting_citations":[{"why":"Supplies the SOTA INER baseline (CPFD) whose feature distillation and pseudo-labeling strategy LGFD extends, and provides the comparison and ablation reference.","marker":"[12]"},{"why":"Provides the ExtendNER distillation-based INER baseline that LGFD compares against.","marker":"[13]"},{"why":"Provides the CFNER causal-distillation baseline and the greedy sampling algorithm used to partition datasets into incremental tasks.","marker":"[48]"},{"why":"Defines the BERT backbone used as the encoder for local models.","marker":"[37]"},{"why":"Supplies the knowledge distillation technique underlying the structural distillation loss.","marker":"[16]"},{"why":"Supplies the federated averaging framework for aggregating local model updates.","marker":"[1]"},{"why":"Provides the I2B2 medical NER dataset used in the main experiments.","marker":"[56]"},{"why":"Provides the OntoNotes5 NER dataset used in the main experiments.","marker":"[57]"}],"fun_headline_variants":["Federated NER that learns new types without forgetting","No forgetting in federated incremental NER","Federated NER: add types, keep old ones","Privacy-safe NER that adapts to new types","LGFD: federated NER with incremental types"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The monitor assumes that the arrival of new entity types always produces a sharp increase in a client's averaged prediction entropy of at least 0.6, so a noisy round or a subtle type change can make it store the wrong old model and break the whole defense.","fun_headline_variants_meta":{"raw":{"variants":["Federated NER that learns new types without forgetting","No forgetting in federated incremental NER","Federated NER: add types, keep old ones","Privacy-safe NER that adapts to new types","LGFD: federated NER with incremental types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1500,"prompt_tokens":909,"completion_tokens":591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":523}},"tokens_in":525,"tokens_out":591,"duration_ms":5727,"temperature":1.0,"reasoning_tokens":523,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:18:25.859509+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a FINER experiment where a new entity type is introduced whose tokens the old model already predicts with low entropy, so the averaged entropy gap never reaches 0.6 while forgetting still occurs; if task switches are missed and final F1 degrades, the monitor assumption is falsified. Alternatively, inject noisy rounds into the local data and check whether false entropy jumps above 0.6 store an outdated teacher and reduce final performance.","supporting_citations":[{"cited_title":"Continual Named Entity Recognition without Catastrophic Forgetting,","cited_arxiv_id":null,"evidence_quote":"Supplies the SOTA INER baseline (CPFD) whose feature distillation and pseudo-labeling strategy LGFD extends, and provides the comparison and ablation reference."},{"cited_title":"Continual learning for named entity recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the ExtendNER distillation-based INER baseline that LGFD compares against."},{"cited_title":"Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the CFNER causal-distillation baseline and the greedy sampling algorithm used to partition datasets into incremental tasks."},{"cited_title":"Distilling the Knowledge in a Neural Network,","cited_arxiv_id":null,"evidence_quote":"Supplies the knowledge distillation technique underlying the structural distillation loss."},{"cited_title":"Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2),","cited_arxiv_id":null,"evidence_quote":"Provides the I2B2 medical NER dataset used in the main experiments."},{"cited_title":"OntoNotes: the 90% solution,","cited_arxiv_id":null,"evidence_quote":"Provides the OntoNotes5 NER dataset used in the main experiments."}],"review_version":1}