{"id":"34dbc4a3-aa1b-4fb4-b3d5-22e93fa28104","arxiv_id":"2606.02857","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"GRZO is a zeroth-order optimizer that applies group-relative loss normalization across per-example perturbations to lower gradient variance, achieving higher accuracy than MeZO with reduced memory on models like Llama3-8B.","lead":"GRZO proposes a group-relative normalization for zeroth-order optimization in LLM fine-tuning, using one perturbation per batch example to increase effective gradient directions without extra memory or forward passes. A smart generalist might read it to understand potential reductions in hardware demands for adapting large AI models.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"GRZO's per-example perturbations cannot be realized at no extra forward cost because parameter perturbations are global to the model","rationale":"The reader's weakest_assumption isolates precisely the point at which the central theoretical and practical claims become internally inconsistent with the mechanics of parameter-space ZO. The empirical numbers cannot be evaluated until that inconsistency is resolved.","tokens_in":1748,"tokens_out":297,"duration_ms":23261,"concrete_test":"Re-implement the GRZO forward step exactly as described in §3 (or the equivalent methods section) and count the number of model forward evaluations per optimization step on a batch of size 8; compare against MeZO on the same hardware. If the count exceeds one batched forward, the 'no additional forward cost' premise is falsified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Standard ZO estimators (including MeZO) apply one perturbation vector v to all parameters, then perform a single batched forward pass on the mini-batch under the perturbed model. Assigning a distinct pseudo-independent v_i to each of the B examples requires either B separate forward passes or an equivalent mechanism that maintains B distinct perturbed parameter sets; neither is compatible with a single forward pass or with inference-level memory. The directional-unbiasedness and variance-shrinkage claims in the proof therefore rest on an implementation that the abstract does not show is possible without violating the stated cost and memory invariants.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes GRZO, a Group-Relative Zeroth-Order optimizer for memory-efficient LLM fine-tuning. It claims to draw one pseudo-independent perturbation per mini-batch example, aggregate losses via group-relative normalization to increase effective gradient directions to the batch size at no extra forward cost, prove directional unbiasedness with variance shrinking proportionally to batch size (yielding tighter nonconvex convergence than MeZO), and report empirical gains: +3.0 average accuracy on Llama3-8B over MeZO at 23% lower peak memory, plus +6.0 average lift to other ZO variants across RoBERTa-large, Llama3-8B, and OPT-13B.","tokens_in":1860,"tokens_out":406,"duration_ms":18400,"significance":"If the implementation and proof hold, GRZO would meaningfully advance ZO fine-tuning by addressing high variance while preserving inference-level memory, potentially broadening applicability of memory-efficient methods to larger models.","major_comments":[{"comment":"Abstract: The central claim that distinct pseudo-independent perturbations can be assigned per mini-batch example 'at no additional forward cost while preserving inference-level memory' requires explicit justification. Standard ZO (including MeZO) applies one global perturbation vector to all parameters for a single batched forward pass; realizing B distinct perturbed parameter sets would appear to necessitate either B forward passes or an equivalent mechanism that increases cost or memory, directly affecting the variance-reduction and convergence claims.","section":"Abstract"},{"comment":"The directional-unbiasedness proof and variance-shrinkage argument (stated in the abstract) rest on the per-example perturbation mechanism; without a concrete description of how this is realized in a single forward pass (e.g., via a specific section or algorithm), the proof cannot be verified as load-bearing for the tighter bound versus MeZO.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the thorough review and for identifying areas where the manuscript's description of the per-example perturbation mechanism requires greater clarity. We address the two major comments below and will revise the paper accordingly to strengthen the presentation of the implementation and supporting analysis.","responses":[{"response":"We agree that the abstract is too concise on this point and does not supply the requested justification. The full manuscript (Section 3.2 and Algorithm 1) outlines the group-relative normalization applied to per-example losses, but the concrete realization of distinct perturbations within a single forward pass is not described with sufficient algorithmic detail. We will revise by expanding Section 3 with a dedicated subsection and updated pseudocode that explicitly shows the computation (including how the batch is processed to achieve the claimed memory and forward-pass cost).","revision_made":"yes","referee_comment":"[Abstract] Abstract: The central claim that distinct pseudo-independent perturbations can be assigned per mini-batch example 'at no additional forward cost while preserving inference-level memory' requires explicit justification. Standard ZO (including MeZO) applies one global perturbation vector to all parameters for a single batched forward pass; realizing B distinct perturbed parameter sets would appear to necessitate either B forward passes or an equivalent mechanism that increases cost or memory, directly affecting the variance-reduction and convergence claims."},{"response":"The referee is correct that the proof in Section 4 relies on the per-example mechanism introduced in Section 3. Because the current description of the single-forward-pass implementation is not sufficiently concrete, the load-bearing relationship between the mechanism and the tighter convergence bound cannot be fully verified from the text. We will revise the manuscript to insert the missing algorithmic details (expanded Algorithm 1 and a short illustrative example) so that the proof can be checked directly against the implementation.","revision_made":"yes","referee_comment":"[—] The directional-unbiasedness proof and variance-shrinkage argument (stated in the abstract) rest on the per-example perturbation mechanism; without a concrete description of how this is realized in a single forward pass (e.g., via a specific section or algorithm), the proof cannot be verified as load-bearing for the tighter bound versus MeZO."}],"tokens_in":1375,"tokens_out":477,"duration_ms":21822,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper's central mechanism does not appear workable as described. Standard zeroth-order methods apply one perturbation vector to the whole model and run one forward pass on the batch. Assigning a distinct perturbation to each example would require either multiple forwards or some equivalent that breaks the single-pass and memory claims. The abstract does not show how they avoid this.\n\nWhat is new is the group-relative normalization step on top of MeZO, plus a claimed proof that this makes the estimator directionally unbiased with variance shrinking by batch size. The experiments run on RoBERTa-large, Llama3-8B, and OPT-13B and report accuracy lifts plus memory savings when used as a drop-in replacement.\n\nThe soft spot is load-bearing. If the per-example perturbations cannot be realized without extra cost, then the directional-unbiasedness proof and the tighter convergence bound rest on an assumption the implementation cannot meet. The reported gains might come from a different practical version, but the abstract and the stress-test note leave that gap open. No other major issues stand out in the reported numbers.\n\nThis is for researchers already working on memory-efficient ZO fine-tuning. A reader in that niche could extract the normalization idea, but the main theoretical and efficiency claims need clarification before they can be used. I would send it to peer review so the authors can either show the implementation or revise the claims; the idea is worth checking even if the current version has this gap.","headline":"GRZO's claim of per-example perturbations at no extra forward cost conflicts with how global ZO perturbations work, so the variance reduction and unbiasedness results do not follow.","tokens_in":2331,"tokens_out":381,"would_cite":false,"duration_ms":29115,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"GRZO reduces variance in zeroth-order LLM fine-tuning by normalizing losses across batch perturbations.","keywords":["zeroth-order optimization","large language model fine-tuning","gradient estimation","memory-efficient optimization","nonconvex convergence","group normalization","variance reduction"],"falsifier":"An experiment that measures the variance of GRZO gradient estimates and finds it does not shrink as batch size grows would disprove the variance reduction and tighter convergence claims.","tokens_in":2652,"feed_emoji":"📉","tokens_out":663,"duration_ms":29348,"temperature":0.7,"pith_summary":"The paper proposes GRZO to overcome the high variance problem in zeroth-order optimization for fine-tuning large language models. By drawing a single pseudo-independent perturbation for each example in a mini-batch and normalizing the resulting losses in a group-relative manner, the method effectively multiplies the number of gradient directions by the batch size. This is achieved without any extra forward passes or memory overhead beyond standard inference. The authors prove the estimator remains directionally unbiased while its variance decreases with larger batches, leading to improved convergence guarantees over previous methods like MeZO. Empirical results show consistent accuracy gains across several model sizes and tasks.","feed_headline":"GRZO reduces ZO gradient variance with batch normalization","feed_subtitle":"The method keeps MeZO memory use while proving tighter bounds and gaining 3 accuracy points on Llama3-8B","key_machinery":"Group-relative normalization applied to per-example zeroth-order loss estimates from individual perturbations","core_discovery":"GRZO draws one pseudo-independent perturbation per mini-batch example and aggregates the per-example losses through group-relative normalization, raising the effective gradient-direction count from one to the batch size at no additional forward cost while preserving inference-level memory. GRZO is directionally unbiased with variance shrinking proportionally to the batch size, yielding a tighter nonconvex convergence bound than MeZO.","pith_inferences":["The group-relative normalization step could apply to variance reduction in other single-forward-pass estimators outside language models.","Larger batches in GRZO may continue to tighten convergence in practice beyond the tested sizes.","Pairing GRZO with additional compression methods might allow fine-tuning of models larger than those evaluated here under fixed hardware limits."],"forward_implications":["GRZO yields a tighter nonconvex convergence bound than MeZO because variance shrinks with batch size.","GRZO raises average accuracy by 3.0 points over MeZO on Llama3-8B while cutting peak GPU memory by 23 percent.","GRZO improves sparse, low-rank, and quantized ZO variants by 6.0 points on average when substituted for the MeZO core.","GRZO preserves inference-level memory use on RoBERTa-large, Llama3-8B, and OPT-13B across multiple tasks."],"fun_headline_variants":["GRZO raises ZO gradient directions to batch size with group normalization","GRZO reduces ZO variance proportionally to batch size","GRZO is directionally unbiased with variance shrinking by batch size","GRZO yields tighter ZO convergence than MeZO on LLMs"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Drawing one pseudo-independent perturbation per mini-batch example and aggregating losses via group-relative normalization raises the effective gradient-direction count from one to the batch size at no additional forward cost while preserving the memory advantage.","fun_headline_variants_meta":{"raw":{"variants":["GRZO raises ZO gradient directions to batch size with group normalization","GRZO reduces ZO variance proportionally to batch size","GRZO is directionally unbiased with variance shrinking by batch size","GRZO yields tighter ZO convergence than MeZO on LLMs"]},"model":"grok-4.3","cost_usd":0.005509,"raw_usage":{"total_tokens":2628,"prompt_tokens":634,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":55087000,"prompt_tokens_details":{"text_tokens":634,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1925,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":634,"tokens_out":69,"duration_ms":14505,"temperature":1.0,"reasoning_tokens":1925,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T15:29:41.824007+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment that measures the variance of GRZO gradient estimates and finds it does not shrink as batch size grows would disprove the variance reduction and tighter convergence claims.","supporting_citations":[],"review_version":1}