{"id":"370cf29b-94a1-49f5-a152-e5a188ae07d5","arxiv_id":"2607.11528","paper_version":3,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"HermesHFL adds incentive contracts and gradient-ascent unlearning to hierarchical federated LoRA fine-tuning for leave-unlearn-rejoin clients, but its claimed superiority is contradicted by its own experiments.","lead":"A proposed system, HermesHFL, combines hierarchical federated learning, LoRA fine-tuning, and payments/penalties to let clients erase their data's influence from an LLM and later rejoin training. The paper claims consistent wins over baselines, but its own tables show two baselines with higher test accuracy, and no existing federated-unlearning method is compared.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported Tables II and III contradict the abstract's 'consistently outperforms' claim: GenSA and GdyRatio achieve higher test accuracy than Neogen on SST-2 in both configurations.","rationale":"The reader's verdict is REJECT with high confidence, and my independent read agrees that the paper should be rejected. However, the reader's designated weakest assumption was the unproven reduction of the three-layer problem P0 to the two-layer P1 via the contribution-proportional budget rule in Eq. (32). That is a valid theoretical concern, but the more decisive, load-bearing issue is empirical: the paper's own Tables II and III show Neogen with lower test accuracy than GenSA and GdyRatio on SST-2 in both Set#1 and Set#2. Since the abstract explicitly claims consistent outperformance in model utility, and the evaluation section treats Acc as the primary model-utility metric, the reported numbers are internally contradictory. This is not a subtle assumption that could be patched with a lemma; it is a direct falsification of the central claim. I therefore focus my stress-test on that contradiction. The agreement_with_reader is 'partial' because the reader did identify the table contradiction in their rationale, but not as the weakest assumption. My concrete test would settle the issue by rerunning the relevant comparisons or by forcing the authors to specify and report the exact utility composite that would make their claim true. The verdict remains REJECT/UNCHANGED because the contradiction is already present in the submitted text; no additional experiment is needed to notice it, though reproduction would confirm it.","tokens_in":38451,"tokens_out":2601,"duration_ms":27311,"concrete_test":"Reproduce the optimization comparison in Tables II and III using the paper's stated deterministic setup: SST-2 and AGNews, LoRA rank 8 on GPT-2, 6 global rounds, 5 edge rounds, Set#1 (10 workers, 3 managers) and Set#2 (20 workers, 4 managers), with Neogen, EAOnly, GenSA, GdyRatio, and RandAssign. Run at least 5 seeds. If GenSA and GdyRatio continue to show higher test accuracy than Neogen on SST-2, the 'consistently outperforms in model utility' claim is false. Alternatively, if the authors intend a composite utility metric, re-evaluate all baselines under that exact metric and show Neogen dominates; otherwise the contradiction stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that HermesHFL 'consistently outperforms state-of-the-art baselines in model utility.' The paper's own optimization comparison tables directly contradict this. In Table II (SST-2, Set#1), Neogen accuracy is 0.795 while GenSA is 0.877, GdyRatio is 0.834, and RandAssign is 0.845. In Table III (SST-2, Set#2), Neogen accuracy is 0.837 while GdyRatio is 0.903, GenSA is 0.890, and RandAssign is 0.879. The text even acknowledges that 'GenSA achieves the highest accuracy on SST-2 Set#1 (0.877).' On AGNews (Table II) Neogen is highest, but 'consistently' fails when the primary model-utility metric, test accuracy, is lower than three baselines on one dataset in both scales. This is an internal, numerical contradiction, not a matter of interpretation. Unless 'model utility' is redefined to exclude accuracy or to incorporate utility fairness in a way that is never formalized, the abstract's central claim is falsified by the paper's own data. This alone is sufficient to reject the paper as written. Other concerns (unproven P0-to-P1 reduction, weak unlearning validation, convergence proof relying on universal approximation) compound the problem, but the internal accuracy comparison is the most load-bearing because it directly refutes the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HermesHFL, a hierarchical federated learning framework with selective unlearning, client rejoining, and incentive mechanisms for LoRA-based LLM fine-tuning. It formulates a three-layer optimization problem (P0) involving president, managers, and workers, then reduces it to a two-layer problem (P1) via a contribution-proportional budget allocation rule. The proposed solver, Neogen, combines CMA-ES for continuous incentive variables and CHC evolutionary search for discrete selection/association decisions, with a neural-network surrogate to accelerate lower-level optimization. Experiments on SST-2 and AGNews with GPT-2 LoRA compare Neogen against generic optimization heuristics and compare three unlearning strategies. The paper claims that HermesHFL \"consistently outperforms state-of-the-art baselines in model utility, unlearning effectiveness, convergence stability, and resource efficiency.\"","tokens_in":38795,"tokens_out":7156,"duration_ms":72134,"significance":"If the claims were fully supported, the paper would make a useful contribution: a unified model of hierarchical federated unlearning with dynamic client leave-unlearn-rejoin behavior, LoRA-based PEFT, and economic incentives is a relevant and timely problem. The formalization of the lifecycle and the integration of a bilevel evolutionary solver with a neural surrogate are interesting. The paper reports reproducible hyperparameters, albeit without releasing code. However, the central claims are not supported by the paper's own numerical results, the problem reduction is assumed rather than proved, and the validation of unlearning effectiveness is weak. These are load-bearing issues for the stated contribution.","major_comments":[{"comment":"The abstract and conclusion claim that HermesHFL \"consistently outperforms state-of-the-art baselines in model utility.\" The evaluation defines Acc as the model-utility metric. Tables II and III contradict this claim: on SST-2 Set#1, Neogen achieves 0.795 versus GenSA 0.877, GdyRatio 0.834, and RandAssign 0.845; on Set#2, Neogen achieves 0.837 versus GdyRatio 0.903, GenSA 0.890, and RandAssign 0.879. The text itself states that \"GenSA achieves the highest accuracy on SST-2 Set#1 (0.877).\" Unless \"model utility\" is redefined to a composite that excludes accuracy or weights it in an unstated way, the central claim is false on the paper's own data. This is a load-bearing numerical contradiction, not a presentation issue.","section":"Sec. VII-B / Tables II-III"},{"comment":"The fixed-pricing baseline is calibrated to Neogen's output: \"the multiplier is set to 3, which corresponds to the midpoint of the p/cost ratio range learned by Neogen.\" This is circular for the incentive-efficiency comparisons. The baselines are not independent \"state-of-the-art\" mechanisms; their contracts are constructed from the proposed method's learned range. As a result, utility comparisons (MgU/WkU) and any downstream claims of incentive-compatible superiority are partially built into the baseline design.","section":"Sec. VII-B"},{"comment":"The transformation of three-layer problem P0 to two-layer problem P1 assumes that the president's budget allocation equals the contribution-proportional rule of Eq. (32). The text calls this a \"presumption,\" and Appendix A supplies only analogies, not a proof. If the true optimal allocation differs from proportional-to-contribution, Neogen solves P1, not P0, and the claimed hierarchical optimum is not preserved. A formal equivalence result, or a counterexample analysis, is necessary for the paper's central optimization claim.","section":"Sec. V-C / Eq. (32)"},{"comment":"The title and abstract claim \"incentive-compatible,\" but the paper provides no mechanism-design analysis. Eqs. (25)-(30) define utilities, and the constraints enforce individual rationality and budget feasibility, yet nothing shows that truthful reporting is a best response, that contracts are incentive-compatible, or that the evolutionary solver finds an equilibrium of the induced game. The term appears to be used descriptively rather than as a verified property.","section":"Sec. V-A/B / Eqs. (25)-(30)"},{"comment":"Unlearning effectiveness is verified only by KLD between pre- and post-unlearning predictions on the erased worker's own data. This shows that gradient ascent changed predictions on that data; it does not show closeness to the retraining benchmark A(D_rtn) required by Eq. (4), nor does it test retained-data utility or leakage via membership inference. The reported KLD values (e.g., 7.29 on SST-2) therefore do not establish \"unlearning effectiveness\" against state-of-the-art baselines, none of which are actual federated-unlearning methods.","section":"Sec. V-A4 / Eq. (24)"},{"comment":"The convergence proof is a sequence of qualitative observations: CHC is elitist and the search space is finite; the NN is a universal approximator; CMA-ES has known convergence properties. These do not imply convergence of the surrogate-assisted bilevel scheme to a stationary solution of P1. CHC solves (41) only approximately, the NN mapping v*=NN(u;theta) has unknown approximation error, and the upper-level fitness is evaluated with approximate lower-level solutions. The claim that Neogen \"converges to a stable stationary solution\" is not established.","section":"Sec. VI-C"}],"minor_comments":[{"comment":"The per-class sample counts are said to be \"explicitly specified per worker\" but are not reported. A reproducibility table for the data splits is needed.","section":"Sec. VII-A"},{"comment":"The abstract claims wall-clock resource efficiency, but the optimization tables report only Feval/r. Wall-clock times are given only for the unlearning-strategy comparison. Clarify which efficiency claim is being made.","section":"Sec. VII-B / Tables II-III"},{"comment":"In the received version, the figure labels and legends appear garbled (e.g., placeholder sequences in Figs. 3-8). The tables in Appendix C are readable; the figures should be regenerated or supplemented with more readable captions.","section":"Figures 3-8"},{"comment":"If the experiments are retained, the wording \"consistently outperforms\" should be replaced with a claim matching the data, e.g., \"competitive on SST-2 and superior on AGNews,\" or provide a formal composite utility that justifies the claimed ordering.","section":"Abstract / Conclusion"}],"recommendation":"reject","confidential_remarks":"The internal numerical contradiction in Tables II-III is decisive for the headline claim, and the P0-to-P1 reduction is assumed rather than proved. The baseline calibration from Neogen's learned range further undermines the comparative claims. I see no simple local revision that preserves the paper's contribution as stated. A future resubmission would need to narrow the claims, add a formal treatment of the hierarchy reduction, and validate unlearning against actual federated-unlearning baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The problem is real and worth working on: selective unlearning with client rejoining in hierarchical federated fine-tuning, with incentives on top. The authors are right that prior work mostly treats flat FL or ignores the leave-unlearn-rejoin lifecycle. The LoRA-based hierarchical aggregation, the reputation score, and the bilevel CMA-ES + CHC + neural surrogate optimizer are all reasonable engineering, and the framework is described in enough detail that someone could reimplement it. That part deserves credit.\n\nThe soft spots are not minor. The abstract claims HermesHFL \"consistently outperforms\" baselines in model utility, but Tables II and III show GenSA at 0.877 and GdyRatio at 0.903 on SST-2 while Neogen gets 0.795 and 0.837. The text even acknowledges GenSA's higher accuracy. That is an internal numerical contradiction, not a matter of interpretation. The claim could be salvaged by redefining utility to include fairness or incentive balance, but the paper never formalizes such a metric.\n\nUnlearning validation is also thin: only KLD on the unlearning client's own data, with no comparison against a retrained gold-standard model and no error bars or seeds. Given that Retrain is only marginally better in accuracy in the unlearning ablation, the authors clearly have the machinery to do the right comparison; they just do not report it for the main unlearning claim.\n\nThe P0-to-P1 reduction in Eq. (32) simply assumes the president allocates budget proportional to worker contribution. If that is a design choice, fine, but it is not justified as preserving the original optimum. The convergence analysis leans on the universal approximation theorem for a finite training set and on CMA-ES convergence results that do not apply to this bilevel non-convex problem. And the fixed-pricing baseline uses a multiplier calibrated to the midpoint of the range Neogen learned, which is circular.\n\nI do not think this is a desk-reject paper. The problem setting is timely, the framework is reusable, and the flaws, while serious, are mostly fixable: tone down the claims, add a real retrain baseline, run multiple seeds, justify or drop the reduction, and replace the circular baseline. But as written, the central claim is falsified by the paper's own data, so I would not accept it or cite it. Send it to referees who can push for that revision, but expect a major overhaul.","headline":"A timely problem and a plausible framework, but the paper's own tables contradict the headline claim: on SST-2, GenSA and GdyRatio beat Neogen on accuracy.","tokens_in":39399,"tokens_out":1432,"would_cite":false,"duration_ms":16914,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HermesHFL claims federated LLM unlearning can be scheduled, funded, and reversed in one optimization framework with no full retraining.","keywords":["hierarchical federated learning","machine unlearning","LLM fine-tuning","LoRA","incentive mechanism","bilevel optimization","client rejoining","evolutionary algorithm"],"falsifier":"Run the exact P0 optimization and Neogen's P1 optimization on a small instance (e.g., 6 workers, 2 managers, 2 budget levels) by exhaustive enumeration, and compare the resulting president utility under the true optimal budget allocation versus the Eq. (32) proportional allocation. If the proportional allocation is not optimal for P0, the paper's central hierarchy-preservation claim fails; likewise, if the optimum differs, the reported accuracy and utility numbers would change.","tokens_in":38226,"feed_emoji":"🧠","tokens_out":5448,"duration_ms":51931,"temperature":0.7,"pith_summary":"This paper sets out to show that hierarchical federated learning (HFL) for large language models can support the full life cycle of client participation — joining, requesting erasure, leaving, and later rejoining — without sacrificing model accuracy or blowing up compute. To do that it couples a three-tier president–manager–worker architecture with a contract mechanism: payments, unlearning penalties, and budget allocations are optimized together with which workers join and which edge server they attach to. The central move is to reduce the three-level optimization to a two-level one by assuming the president distributes budget in proportion to each manager's workers' contributions, then solving that problem with a bilevel evolutionary optimizer (CMA-ES for continuous incentives, CHC for binary decisions) accelerated by a neural surrogate. The paper's experiments on GPT-2 LoRA fine-tuning over SST-2 and AGNews claim that this joint design beats four baselines in accuracy, utility balance, unlearning fidelity, convergence stability, and fitness-evaluation cost. If correct, the framework would make erasure requests, departures, and re-entry routine events rather than disruptions in federated LLM training.","feed_headline":"Selective erasure keeps federated LLMs accurate at half retraining cost","feed_subtitle":"Jointly optimizing payments, penalties, and client selection makes leave-unlearn-rejoin routine and budget-neutral.","key_machinery":"The load-bearing identity is the contribution-proportional budget rule, Eq. (32): manager j's new budget equals the global budget times the share of total training contribution theta produced by j's selected workers. This turns the three-layer president–manager–worker problem P0 into the two-layer manager–worker problem P1, because the president's budget decision becomes a deterministic function of lower-level choices. Neogen then solves P1 by nesting CMA-ES over continuous payments and penalties (upper level) with CHC over binary participation and association variables (lower level), using a neural network to approximate the mapping from upper-level solutions to lower-level optima so that C","core_discovery":"On the paper's own terms, the core discovery is that selective unlearning does not need a separate retraining pipeline: unlearning can be executed inside the same global aggregation round as normal training, by having clients with erasure requests run gradient ascent on their LoRA parameters while the rest run stochastic gradient descent, then aggregating both update types at edge and global levels. The KLD between pre- and post-unlearning output distributions over the departing worker's data verifies erasure; once that KLD exceeds a threshold, the worker's profile is refreshed and it may rejoin. Around this loop the paper wraps an economic layer: each manager signs contracts specifying paym","pith_inferences":["An untested consequence of Eq. (32) is that budget allocation is artificially tied to the current proxy theta for contribution; if quality estimates are noisy, managers with temporarily lucky workers receive budget that a directly optimizing president would not give them. One could test this by comparing P1 against an exhaustive small-grid solution of P0.","The framework assumes a worker's cost and privacy cost are known to the manager when signing contracts; if workers can misreport these costs, incentive compatibility may fail. A natural extension would be to add truthful-reporting constraints or a screening contract menu.","The KLD threshold delta is a verification heuristic, not a privacy guarantee; a committed adversary could in principle retain information that does not shift output distributions. A stronger erasure test would combine KLD with membership-inference or gradient-residual checks.","If this design scales to larger models, the same contribution-proportional budget rule could be paired with Shapley-style contribution measures to make budget shares more robust to collusion among workers under one manager."],"forward_implications":["If the central claim holds, a federated LLM can honor erasure requests and still keep a single continuously updated global model; no full retraining is needed after each request.","Unlearning penalties recycled into manager budgets turn client departures from a cost into future recruitment capacity, making churn economically tolerable.","Because worker profiles are refreshed after verified erasure, a rejoining client cannot silently re-inject the data it asked to erase.","The NN surrogate's approximation of lower-level optima reduces per-round fitness evaluations by about 25% compared to the surrogate-free evolutionary baseline, which matters when each evaluation involves LLM fine-tuning.","Balanced positive utilities for both workers and managers across dataset scales suggest the contract design can sustain participation in regimes where fixed-pricing baselines collapse into negative manager utility."],"fun_headline_variants":["Unlearn in the same round: federated LLMs drop data, not accuracy","Leave, unlearn, rejoin: HermesHFL makes erasure part of training","Gradient ascent deletes data while federated training continues","Pay per erasure: incentive-compatible unlearning for LLM fine-tuning","Same-round unlearning: federated LLMs forget on demand"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The conclusion rests on the assumption that the contribution-proportional budget allocation in Eq. (32) exactly preserves the optimum of the original three-level problem; if the president's true optimal allocation differs from that rule, Neogen solves a different problem than P0 and the reported gains do not transfer to the hierarchical setting.","fun_headline_variants_meta":{"raw":{"variants":["Unlearn in the same round: federated LLMs drop data, not accuracy","Leave, unlearn, rejoin: HermesHFL makes erasure part of training","Gradient ascent deletes data while federated training continues","Pay per erasure: incentive-compatible unlearning for LLM fine-tuning","Same-round unlearning: federated LLMs forget on demand"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000999,"raw_usage":{"total_tokens":4064,"prompt_tokens":742,"completion_tokens":3322,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":3224}},"tokens_in":486,"tokens_out":3322,"duration_ms":21604,"temperature":1.0,"reasoning_tokens":3224,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:20:33.019607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact P0 optimization and Neogen's P1 optimization on a small instance (e.g., 6 workers, 2 managers, 2 budget levels) by exhaustive enumeration, and compare the resulting president utility under the true optimal budget allocation versus the Eq. (32) proportional allocation. If the proportional allocation is not optimal for P0, the paper's central hierarchy-preservation claim fails; likewise, if the optimum differs, the reported accuracy and utility numbers would change.","supporting_citations":[],"review_version":3}