{"id":"09ff14c0-9363-454f-965b-0aba7fd6c3de","arxiv_id":"2508.06467","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"GRIN proposes a gradient-ratio-based metric to find memorization-related parameters, injects noise into them, and then fine-tunes to achieve targeted LLM unlearning.","lead":"This paper introduces GRIN, a method for making LLMs forget specific data by first identifying the parameters most responsible for that data and then adding noise to them before fine-tuning. If it works, it could make machine unlearning more precise and less damaging to a model's other knowledge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gradient-ratio localization is the load-bearing premise; abstract provides no evidence that high gradient ratio identifies memorization parameters rather than loss-scale artifacts.","rationale":"The reader identified the same load-bearing assumption: that the gradient ratio is a reliable indicator of parameter localization. I agree that this is the crux. However, the reader framed it as an abstract-only limitation; I am adding a concrete technical pathway by which the assumption could fail (loss-scale artifacts, distributed representation) and proposing a targeted experimental test that would adjudicate it. Since the manuscript is abstract-only, we cannot determine whether the full text already includes such a validation, so the verdict should remain UNVERDICTED as the reader set it. My concern does not move the verdict; it sharpens the exact question the full text must answer.","tokens_in":575,"tokens_out":2381,"duration_ms":32121,"concrete_test":"On a small transformer (e.g., GPT-2), construct a controlled forget/retain split where the forget set contains known memorized facts (e.g., specific name-birthdate pairs). Compute GRIN's gradient-ratio scores over parameters. Independently localize the parameters and directions that actually mediate these facts using causal tracing (activation patching) to identify the minimal parameter subset that, when ablated, causes forgetting. Compare the overlap between the GRIN-selected top-k parameters and the causal-tracing localization. If the overlap is no better than random selection, the localization premise fails; if overlap is high, the concern is addressed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that selective noise injection into parameters identified by gradient ratio improves unlearning while preserving utility hinges on the premise that the gradient ratio between forget and retain data actually localizes memorization. This premise is not self-evident. Gradient ratios are a first-order, local signal evaluated at a specific parameter state; they can be dominated by differences in loss scale, gradient noise, or near-zero retain gradients, rather than by parameters causally responsible for storing specific knowledge. In a distributed model, memorization is often spread across many parameters and directions, so a top-k selection of high-ratio parameters may either miss the relevant circuitry or select parameters that, when perturbed, damage general capabilities. If the localization is weak, noise injection degrades utility without achieving forgetting, and the method collapses to random-parameter noise injection with fine-tuning. The abstract reports only benchmark validation, with no ablations, no comparison to random parameter selection, and no quantitative results. Thus the core premise is entirely unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes GRIN, an LLM unlearning framework that uses a gradient-ratio-based metric to identify parameters most responsible for memorizing forget data, injects noise into those parameters prior to fine-tuning, and introduces new evaluation metrics. The abstract claims improved unlearning performance and preserved model utility, validated on TOFU, WMDP, and SafePKU. The available text is abstract-only; no quantitative results, definitions of the proposed metrics, or ablations are provided in the reviewable material.","tokens_in":816,"tokens_out":2389,"duration_ms":30392,"significance":"If the claimed results hold, GRIN would add a parameter-localization signal to the LLM unlearning toolbox and a set of tailored evaluation metrics, with potential practical relevance for compliance-driven forgetting. The approach is falsifiable and benchmark-oriented, and the gradient-ratio idea is worth testing. However, the current manuscript as available provides no empirical evidence, no comparison to existing methods, and no validation of the new metrics, so the significance cannot be assessed beyond the plausibility of the concept.","major_comments":[{"comment":"The central premise—that the ratio of forget-data gradients to retain-data gradients identifies parameters that store memorized information—is asserted without supporting evidence. No ablation compares top-k gradient-ratio selection to random parameter selection, loss-scale-matched noise, or alternative attribution methods (e.g., influence functions, TracIn). Without such a control, the claimed improvement could arise from fine-tuning alone or from noise affecting utility in an unmeasured way. Please report this ablation and related baselines.","section":"Abstract"},{"comment":"The claim that GRIN 'improves unlearning performance while maintaining model utility' is unsupported by any quantitative result in the available text. No effect sizes, benchmark scores, baseline comparisons, error bars, or significance tests are reported. A central empirical claim of this strength requires at least a table comparing GRIN against existing unlearning methods on TOFU, WMDP, and SafePKU, with forget-quality and utility metrics reported together.","section":"Abstract"},{"comment":"The manuscript introduces 'new evaluation metrics tailored to the LLM setting' but does not define them or demonstrate their validity. If these metrics are used to support the main claim, the review cannot judge whether they measure forgetting and utility as intended or whether they are co-adapted to the method. Please specify the metrics formally and report them alongside established metrics (e.g., TOFU forget quality, WMDP accuracy drop, retain-set perplexity) to rule out circularity.","section":"Abstract"}],"minor_comments":[{"comment":"The term 'modular' is used but not defined; clarify which stages of GRIN (localization, noise injection, fine-tuning) are independently replaceable.","section":"Abstract"},{"comment":"The framing around 'legal and ethical scrutiny' and 'unauthorized data' may imply compliance guarantees that the method cannot provide; suggest softening to 'may assist in' or adding caveats.","section":"Abstract"},{"comment":"The acronym GRIN is introduced but not expanded; spell out 'Gradient Ratio-based Influence estimation and Noise injection' for clarity.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This report is based solely on the abstract, as the full text was not provided for review. The recommendation of major revision reflects the absence of empirical support in the reviewable material; if the full manuscript contains the missing experiments, ablations, and metric definitions, the major concerns may be largely resolved. I also note that the proposed evaluation metrics should be carefully validated against existing benchmarks before publication, to avoid metric-based circularity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked for my take on this one. It's an abstract-only review, so everything is provisional, but here's where I land. The headline claim is that GRIN uses a gradient-ratio-based metric to pick parameters that store forget-data, injects noise into those, then fine-tunes, and this beats existing unlearning methods on TOFU, WMDP, and SafePKU without killing utility. That's a clean, concrete idea, and the modular framing is sensible. The proposed new evaluation metrics for LLM unlearning are also a reasonable thing to push for. So the paper at least asks the right question and proposes a testable mechanism. That counts for something. \n\nThe soft spot is the load-bearing premise. The abstract gives no quantitative results, no ablations, no comparison to random parameter noise, and no error bars. The stress-test note is right that gradient ratio is a local, first-order signal that can be dominated by loss-scale differences or near-zero retain gradients. It might localize memorization, or it might just identify parameters where the forget loss happens to be high for uninteresting reasons. If that localization is weak, the method collapses to random noise injection plus fine-tuning, and the benchmark gains wouldn't mean much. The paper also doesn't show that the new metrics aren't tailored to favor GRIN. These are not fatal flaws on their face—they're the kind of thing the full paper could answer—but the abstract doesn't give us enough to judge. \n\nI'd send this to peer review, but not for a quick accept. The referee packet should explicitly ask for: an ablation replacing the gradient-ratio selection with random parameter selection, a comparison against a plain fine-tuning baseline, and effect sizes with variance across seeds. If those are in the full text, great; if not, the empirical claim is unsupported. The idea is novel enough to merit a look, and the benchmark choices are standard. Just don't let it through without evidence that the localization signal is real. \n\nFor a reading group, I'd hold until a full version is available. I wouldn't cite it yet—there's nothing to point to. Serious thinker: unclear from the abstract alone; the logic is coherent, but I can't see how deeply they engaged with prior influence-function work or gradient-based localization methods.","headline":"Plausible method, but the core gradient-ratio localization premise is unverified and the abstract gives no numbers—worth a referee, but only with a demand for ablations and random baselines.","tokens_in":1210,"tokens_out":1078,"would_cite":false,"duration_ms":15028,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Gradient ratios locate where an LLM memorizes data, then noise erases only those parameters.","keywords":["machine unlearning","large language models","gradient ratio","noise injection","influence estimation","TOFU","WMDP","SafePKU"],"falsifier":"A direct test would be to run GRIN on a benchmark where ground-truth memorized parameters are known (for example, by inserting a known fact and tracing which parameters change), and check whether the highest-gradient-ratio parameters overlap those true stores; a second test would compare GRIN against injecting the same noise magnitude into randomly chosen parameters—if random noise performs equally, the gradient-ratio localization is not doing the work.","tokens_in":528,"feed_emoji":"🎯","tokens_out":1452,"duration_ms":17554,"temperature":0.7,"pith_summary":"This paper claims that machine unlearning in large language models can be made precise by first identifying the specific parameters that store the information to be forgotten. It introduces GRIN, a framework that uses a gradient-ratio-based influence metric to localize memorized knowledge, then injects selective noise into those parameters before fine-tuning. The authors argue this targeted approach improves forgetting while preserving the model's general utility, and they validate it on the TOFU, WMDP, and SafePKU benchmarks. A sympathetic reader would care because it offers a path to unlearning that is less destructive to unrelated knowledge than existing methods.","feed_headline":"Gradient ratios point to exactly which LLM parameters hold the data to forget","feed_subtitle":"Targeted noise on those parameters erases the data while preserving the model's other knowledge.","key_machinery":"The key mechanism is the gradient-ratio metric: for each parameter, the ratio of its gradient contribution on the forget set to its gradient contribution on the retain set. A high ratio is taken as evidence that the parameter stores forget-set information. That score drives selective noise injection prior to fine-tuning, so the model forgets where it memorized without broad disruption.","core_discovery":"The central claim is that comparing gradients computed on the forget set against gradients on the retain set reveals which parameters are disproportionately responsible for memorizing the forget data. GRIN uses this gradient ratio as a parameter-level influence score, selects the parameters with the highest scores, and injects controlled noise into those parameters before fine-tuning on the retain set. The paper asserts that this selective noise injection improves unlearning performance and maintains model utility, and it also introduces new evaluation metrics tailored to LLM unlearning. This is a method paper: the discovery is the localization metric itself and the demonstration that target","pith_inferences":["The gradient-ratio localization idea could be transferred beyond unlearning, for example to identify which parameters encode biases or harmful behaviors, enabling targeted interventions.","Because the metric is computed from gradients alone, it may scale to very large models more cheaply than activation-based or representation-based localization methods.","A natural testable extension would be to compare GRIN's selected parameters against those identified by probing classifiers or influence functions, to see if they converge on the same regions."],"forward_implications":["If gradient ratios reliably locate memorized parameters, unlearning can be performed with much less collateral damage to general knowledge.","The method is modular: it plugs into existing fine-tuning-based unlearning pipelines as a preprocessing localization step.","The new evaluation metrics could become a standard way to measure whether an LLM has truly forgotten data while retaining utility.","Targeted noise injection may allow iterative or partial forgetting, removing one concept without a full retrain."],"supporting_citations":[],"fun_headline_variants":["Gradient ratios show which LLM neurons hold the data to forget","Targeted noise erases LLM data without wrecking other knowledge","New metric finds the exact parameters behind LLM memorization","Selective noise injection improves LLM unlearning precision"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the gradient ratio between forget and retain data reliably indicates which parameters actually store the information to be forgotten; if that localization signal is weak, the noise injection will either fail to cause forgetting or will damage unrelated capabilities.","fun_headline_variants_meta":{"raw":{"variants":["Gradient ratios show which LLM neurons hold the data to forget","Targeted noise erases LLM data without wrecking other knowledge","New metric finds the exact parameters behind LLM memorization","Selective noise injection improves LLM unlearning precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":939,"prompt_tokens":617,"completion_tokens":322,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":361,"completion_tokens_details":{"reasoning_tokens":252}},"tokens_in":361,"tokens_out":322,"duration_ms":3726,"temperature":1.0,"reasoning_tokens":252,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:40:36.898125+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to run GRIN on a benchmark where ground-truth memorized parameters are known (for example, by inserting a known fact and tracing which parameters change), and check whether the highest-gradient-ratio parameters overlap those true stores; a second test would compare GRIN against injecting the same noise magnitude into randomly chosen parameters—if random noise performs equally, the gradient-ratio localization is not doing the work.","supporting_citations":[],"review_version":1}