{"id":"2375b030-193c-46a1-819f-23d9679b3e80","arxiv_id":"2506.02230","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SISA++ uses weight averaging of shard-trained models to outperform SISA in retaining SER and DD performance after user-data unlearning on CREMA-D and E-DAIC.","lead":"This paper applies machine unlearning to speech emotion recognition and depression detection, proposing SISA++, which averages models trained on data shards to forget user data while keeping performance. It finds that TRILLsson features with a transformer downstream retain accuracy best after unlearning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central unlearning claim is unsupported: Table 2 only shows utility retention, and no membership-inference or retrain-from-scratch comparison verifies that removed users' influence is actually gone from SISA++'s weight-averaged model.","rationale":"The reader's weakest assumption is exactly the gap I would stress: the experiments measure utility retention, not forgetting. My independent reading of Section 3.2 and Algorithm 1 confirms that no forgetting metric is present, and the weight-averaging mechanism makes the utility result even harder to interpret as unlearning-specific. I do not see an internal inconsistency in the utility comparison itself; the reported numbers are plausible. The decisive missing piece is verification that the deleted users' influence is removed, which would require MIA or equivalence to retraining from scratch. Because the reader already made the verdict conditional on exactly this evidence, my stress-test does not move the verdict; it specifies the concrete test that would settle it. If the proposed MIA/closeness check fails, the verdict should move to REJECT, because the central claim would then be only about utility preservation, not unlearning.","tokens_in":9777,"tokens_out":4683,"duration_ms":45324,"concrete_test":"Run a membership inference attack (e.g., LiRA) on the deleted users' utterances for the 8-shard, 1-user and 2-user removal conditions of SISA and SISA++, and compare MIA success against (a) a pre-unlearning full-data model and (b) a model retrained from scratch on the retained data. Also compute a closeness metric between SISA++ and the retrain-from-scratch model on the forgotten samples (e.g., mean KL divergence of output distributions). If SISA++ MIA success is close to the full-data model or substantially above the retrain-from-scratch level, the central 'unlearning' claim fails regardless of Table 2 utility numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's strongest claim—'SISA++ consistently outperforms SISA, demonstrating superior retention of performance post-unlearning'—rests on the assumption that utility preservation after retraining an affected shard is evidence of unlearning. Every row of Table 2 reports only accuracy/F1/MAE/RMSE on a test set; no membership inference attack (MIA), no closeness-to-retrained-model metric, and no retrain-from-scratch baseline is reported. Since Algorithm 1 obtains the final model by averaging weights across shard models, and weight averaging is independently known (model soups) to improve accuracy, the observed gains may reflect that averaging effect rather than successful removal of data influence. More importantly, if the retrained shard model still memorizes deleted users' utterances—possible when retraining starts from the same weights or is early-stopped—the method would not satisfy the 'right to be forgotten' framing in Section 1 even though utility is retained. Thus the load-bearing, unverified condition is that post-unlearning utility preservation implies actual forgetting; the paper provides no evidence for it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SISA++, a modification of SISA for machine unlearning in paralinguistic speech processing. SISA++ trains one model per data shard and merges the shard models by weight averaging (Algorithm 1) instead of aggregating their outputs at inference. Experiments on CREMA-D (speech emotion recognition) and E-DAIC (depression detection) compare SISA and SISA++ under one- and two-user removal with four-shard and eight-shard configurations, across five feature sets (MFCC, x-vector, XLS-R, WavLM, TRILLsson) and three downstream networks (SVM, CNN, Transformer). The paper reports that SISA++ retains accuracy/F1/MAE/RMSE better than SISA after unlearning, and it distills the findings into 'cookbook recipes' that recommend TRILLsson features with a transformer encoder for best post-unlearning utility retention.","tokens_in":9973,"tokens_out":4263,"duration_ms":40948,"significance":"If the utility-retention result is robust, SISA++ is an appealingly simple, zero-inference-cost modification: it replaces output aggregation with weight averaging and requires no additional training. The paper's strengths include its broad experimental sweep (five feature sets, three downstream models, two datasets, two shard counts, two deletion sizes), the public code and model repository link, and the actionable recommendations for practitioners. The contribution would be a useful empirical result for the paralinguistic speech community, provided the unlearning claim is actually verified; in its current form the evaluation is utility-only and does not establish that deleted users' influence is removed from the model.","major_comments":[{"comment":"The paper's central claim that SISA++ demonstrates superior retention 'post-unlearning' is based entirely on utility metrics; there is no forgetting metric, no membership-inference attack, and no comparison with a retrain-from-scratch model. Because Algorithm 1 forms the final model by averaging all shard-model weights, the observed gains could in principle be the model-soup effect of weight averaging (Ref. [20]) rather than evidence that deleted users' influence has been removed. This distinction matters for the right-to-be-forgotten framing in Section 1; please add at least one verification of forgetting (e.g., membership-inference success before and after unlearning, or closeness to a model retrained without the deleted users), or explicitly reframe the contribution as utility-preserving shard retraining.","section":"§3.2, Table 2"},{"comment":"All comparisons in Table 2 are single-run point estimates without error bars, multiple seeds, or significance tests, yet the text states that SISA++ 'consistently outperforms' SISA. Differences in several cells are small (for example, E-DAIC, two-user removal, eight shards, TRILLsson+Transformer: MAE 4.45 for SISA versus 4.30 for SISA++), and without variance estimates that claim is not supported. Please report means and standard deviations over multiple runs, and use a paired significance test or confidence intervals where the comparison is made.","section":"§3.2, Table 2"},{"comment":"The evaluation protocol is underspecified. It is not stated how a removed user's utterances are assigned to shards (for example, whether all clips of a user stay in one shard), whether the test set contains utterances from removed users, how SISA's output aggregation is implemented for the regression task in Table 2, or whether SISA++ is evaluated with the weight-averaged model MA on the same test set. These details are needed to interpret the comparison and to reproduce the numbers; without them, the observed 4-shard versus 8-shard and SISA versus SISA++ differences are difficult to attribute to the proposed mechanism.","section":"§2.2, §3.2"}],"minor_comments":[{"comment":"The caption contains typos: 'Evalution Scores' should be 'Evaluation Scores', and 'marco average F1 score' should be 'macro average F1 score'.","section":"Table 1 caption"},{"comment":"The third bullet contains 'guideline swith'; it should read 'guidelines with'.","section":"Contributions bullet list"},{"comment":"The sentence 'one notable issue in this context is the is the principle of the right to be forgotten' has a duplicated phrase and should be corrected.","section":"Introduction"},{"comment":"The downstream networks are described as SVM, CNN, and Transformer, but the parameter counts refer to 'FCN models'; please clarify whether FCN is a fourth architecture or a typo for CNN.","section":"§2.3"},{"comment":"The final paragraph contains the raw LaTeX command 'textbfSISA++' instead of formatted bold text.","section":"§3.2"},{"comment":"The phrase 'an novel extension' should be 'a novel extension'.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is aimed at a speech-processing audience, and the missing forgetting verification is the main obstacle to acceptance. The utility-retention comparison has clear practical value, and the public code repository is a positive sign for reproducibility. I would encourage the editor to request a revision that adds a forgetting metric or retrain-from-scratch baseline and reports variance, rather than rejecting outright, because the core empirical contribution is potentially useful even if the unlearning framing must be narrowed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading as the first attempt to bring machine unlearning to paralinguistic speech processing, but the headline contribution is a small tweak—weight averaging across SISA shards—and the evaluation only shows utility retention, not actual forgetting.\n\nWhat is genuinely new: nobody has applied MU to SER/DD before. The paper runs a broad sweep over five feature extractors (x-vector, XLS-R, TRILLsson, WavLM, plus MFCC) and three downstream architectures, and the \"cookbook recipes\" are practical. Code and models are public. The empirical observation that TRILLsson + Transformer is most robust to shard retraining is useful for anyone building privacy-aware PSP systems.\n\nThe main weakness is the gap between the \"unlearning\" label and the evidence. Table 2 reports only accuracy/F1/MAE/RMSE after retraining the affected shard. There is no membership-inference attack, no comparison to a retrain-from-scratch model, no closeness-to-retrained-model metric. So the paper does not demonstrate that removed users' influence is actually gone; it demonstrates that utility is preserved. That would be fine with modest language, but the intro and conclusion claim \"right to be forgotten\" support without that check. Also, the observed gain of SISA++ over SISA may come in part from the known model-soup benefit of weight averaging, which has nothing to do with forgetting. There are no error bars or significance tests, so we can't tell if the consistent superiority is meaningful. Minor: the \"novel extension\" framing overstates what is essentially a one-line change to inference.\n\nThese soft spots are real but they don't kill the paper's usefulness. The core empirical finding—that weight-averaged shard models retain utility better than majority vote after shard retraining on these two datasets—is likely to hold; it just needs to be re-labeled as \"utility-preserving unlearning\" and backed by forgetting metrics in a revision.\n\nSend it to review. A serious referee should ask for the missing baselines and a forgetting metric; with those, this becomes a decent contribution to the privacy-for-speech niche. I would not cite it in its current form, but I'd bring it to a reading group if you work on speech privacy.","headline":"First MU-for-PSP paper; SISA++ is weight-averaged SISA with plausible utility gains, but the forgetting claim itself is unverified.","tokens_in":10531,"tokens_out":2225,"would_cite":false,"duration_ms":19369,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that weight-averaging the shard models of SISA's unlearning pipeline preserves more speech-task accuracy after user-data removal than SISA's vote-based aggregation.","keywords":["machine unlearning","paralinguistic speech processing","speech emotion recognition","depression detection","SISA++","weight averaging","TRILLsson","right to be forgotten"],"falsifier":"Run SISA++ and SISA on identical shard configurations and measure a membership inference attack or parameter distance to a retrained-from-scratch model for the removed users; if SISA++ does not reduce the deleted users' influence relative to SISA, the utility-preservation result does not demonstrate unlearning.","tokens_in":9608,"feed_emoji":"🎙️","tokens_out":5507,"duration_ms":45252,"temperature":0.7,"pith_summary":"This paper brings machine unlearning—selectively removing the influence of a user's data from a trained model—to paralinguistic speech tasks, where voice samples carry emotion and mental-health information. It proposes SISA++, a variant of the SISA unlearning method that merges separately trained data-shard models by averaging their weights instead of combining their outputs by vote. The paper's claim is that this one change consistently preserves more downstream accuracy on speech emotion recognition (CREMA-D) and depression detection (E-DAIC) after simulated one- or two-user removal, across 4- and 8-shard setups, without adding inference cost. It also reports recipe-style guidance, identifying TRILLsson features with a transformer head as the combination that loses least performance after unlearning. If the claim holds, weight averaging is a cheap way to make right-to-be-forgotten requests less damaging to speech models.","feed_headline":"Weight averaging keeps speech models accurate after unlearning","feed_subtitle":"Merging shard weights cuts post-erasure accuracy loss on emotion and depression voice benchmarks.","key_machinery":"The central object is SISA++, the weight-averaging aggregation step added to the SISA unlearning pipeline. After shard sub-models are retrained following a deletion, SISA++ computes $M_A = \\frac{1}{N}\\sum_{i=1}^{N} M_i$, the element-wise average of the shard model weights, and uses that single model for all inference. This replaces SISA's majority-vote or output-averaging aggregation and is the mechanism the paper credits for preserving post-unlearning performance.","core_discovery":"The paper's central claim is that SISA++ outperforms SISA at retaining task performance after unlearning. SISA divides the training set into disjoint shards, trains a sub-model on each, and when deletion requests arrive retrains only the shards containing removed data, combining all sub-model outputs by majority vote (classification) or averaging (regression). SISA++ keeps the sharding and retraining procedure but replaces the output-level combination with a weight-averaged model: the final weights are the arithmetic mean of all shard sub-model weights. Across one-user and two-user removal simulations on CREMA-D and E-DAIC, the paper reports higher accuracy and F1 for emotion recognition and lower MAE and RMSE for depression detection with SISA++ than with SISA in the tested configurations, and it attributes this to the consolidating effect of weight averaging.","pith_inferences":["A follow-up the paper leaves untested: whether weight averaging actually removes the influence of deleted users, or only preserves utility. A membership-inference attack on supposedly unlearned speakers would settle this.","The result hints at a broader principle for shard-based unlearning: parameter-space aggregation can be more lossless than output-space ensembling, which could be tested on image and text benchmarks.","A practical extension of the cookbook recipes would turn the feature-architecture ranking into a compliance checklist, but regulatory value depends on forgetting guarantees, not accuracy retention alone."],"forward_implications":["If SISA++ works as claimed, an unlearning request can be served by retraining only the affected shard and re-averaging weights, with no increase in inference cost over a single model.","Post-unlearning utility on emotion recognition and depression detection is higher with weight averaging than with vote-based aggregation in the configurations tested.","The reported evaluations suggest TRILLsson features paired with a transformer downstream suffer the smallest performance drop after unlearning among the feature-architecture pairs tried.","Finer-grained sharding (8 shards) sometimes reduces degradation on its own, but the SISA++ advantage holds across both 4- and 8-shard settings."],"supporting_citations":[{"why":"Defines SISA, the shard-then-retrain unlearning method that SISA++ extends and the baseline it must beat.","marker":"[7]"},{"why":"Supplies the weight-averaging technique used to merge shard models in SISA++.","marker":"[20]"},{"why":"Shows weight averaging works for speech models, supporting its use in SISA++.","marker":"[21]"},{"why":"Introduces TRILLsson features, which the cookbook recipes identify as the best-performing feature set.","marker":"[22]"},{"why":"Provides the CREMA-D speech emotion recognition benchmark used for evaluation.","marker":"[30]"},{"why":"Provides the E-DAIC depression detection benchmark used for evaluation.","marker":"[31]"}],"fun_headline_variants":["Weight-averaged unlearning retains speech model accuracy","SISA++: weight averaging boosts post-unlearning retention","Merging shard weights cuts unlearning performance loss","New unlearning method keeps speech emotion models sharp","Weight averaging preserves speech models after data removal"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusion stands on the assumption that test accuracy and error after deletion are enough to show unlearning worked; it never measures whether the supposedly deleted users' data still influences the model.","fun_headline_variants_meta":{"raw":{"variants":["Weight-averaged unlearning retains speech model accuracy","SISA++: weight averaging boosts post-unlearning retention","Merging shard weights cuts unlearning performance loss","New unlearning method keeps speech emotion models sharp","Weight averaging preserves speech models after data removal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000626,"raw_usage":{"total_tokens":2850,"prompt_tokens":852,"completion_tokens":1998,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":1923}},"tokens_in":468,"tokens_out":1998,"duration_ms":13046,"temperature":1.0,"reasoning_tokens":1923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:26:57.036586+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SISA++ and SISA on identical shard configurations and measure a membership inference attack or parameter distance to a retrained-from-scratch model for the removed users; if SISA++ does not reduce the deleted users' influence relative to SISA, the utility-preservation result does not demonstrate unlearning.","supporting_citations":[{"cited_title":"Privacy-preserving speech emotion recognition through semi-supervised federated learning,","cited_arxiv_id":null,"evidence_quote":"Defines SISA, the shard-then-retrain unlearning method that SISA++ extends and the baseline it must beat."},{"cited_title":"Privacy risks in speech emotion recognition: A systematic study on gender inference at- tack,","cited_arxiv_id":null,"evidence_quote":"Supplies the weight-averaging technique used to merge shard models in SISA++."},{"cited_title":"Assessing privacy risks of attribute inference attacks against speech-based depression detection system,","cited_arxiv_id":null,"evidence_quote":"Shows weight averaging works for speech models, supporting its use in SISA++."},{"cited_title":"Self-reported symptoms of depression and ptsd are associated with reduced vowel space in screening interviews,","cited_arxiv_id":null,"evidence_quote":"Introduces TRILLsson features, which the cookbook recipes identify as the best-performing feature set."},{"cited_title":"X-vectors: Robust dnn embeddings for speaker recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the CREMA-D speech emotion recognition benchmark used for evaluation."},{"cited_title":"X- vectors meet emotions: A study on dependencies between emotion and speaker recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the E-DAIC depression detection benchmark used for evaluation."}],"review_version":1}