Pith. sign in

REVIEW 4 major objections 7 minor 24 references

OAT-Rephrase: Optimization-Aware Training Data Rephrasing for Zeroth-Order LLM Fine-Tuning

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rephrasing training data with an LLM that has read the MeZO paper consistently improves zeroth-order fine-tuning, often closing the gap with first-order methods.

desk verdict A plausible data-rephrasing trick for MeZO with a genuinely strong paired-run design, but the 'optimization-aware' mechanism is unverified and single-seed results can't support 'consistent'. read the letter →

arxiv 2506.17264 v1 pith:J5KGVSYF submitted 2025-06-10 cs.LG cs.AI

Jikai Long , Zijian Hu , Xiaodong Yu , Jianwen Xie , Zhaozhuo Xu This is my paper
classification cs.LGcs.AI
keywords zeroth-orderoptimizationMeZOLLMfine-tuningtrainingdatarephrasingdata-centricsyntheticmemory-efficientrejection-gatedpipeline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OAT-Rephrase claims that zeroth-order fine-tuning of large language models can be improved by rephrasing the training data before optimization begins. The proposed pipeline has an LLM summarize the MeZO paper, a rewriter LLM rewrite each training instance to be more compatible with MeZO's forward-pass-only gradient estimates, and a judge LLM reject any rewrite that changes the meaning. Evaluated across five classification tasks and three LLM backbones, MeZO fine-tuned on the rephrased data beats MeZO on the original data in 14 of 15 model-task pairs, and often matches or exceeds first-order full fine-tuning and LoRA. If correct, a one-time, model-agnostic data preprocessing step can substitute for some of the algorithmic machinery that zeroth-order methods lack.

What carries the argument

The central object is a rejection-gated rewriting pipeline. A rewriter LLM, GPT-4o, is prompted with a distilled summary of the MeZO paper -- including its abstract, introduction, and paraphrases of its figures and algorithm -- and is instructed to rewrite each training instance to be better suited for MeZO while preserving original meaning. A separate LLM judge checks every rewrite for semantic and logical fidelity, and a rejection gate discards any rewrite that fails, replacing it with the original instance. The rewriter prompt is built through five design phases, including a few-shot instruction module, and the judge prompt is refined until it reaches at least 90% agreement with human judgments on a forty-pair test set.

What would settle it

Measure the empirical variance of MeZO's directional gradient estimates, or the spread of loss values, on original versus OAT-Rephrase training instances at matched parameter states; if variance is not lower on the rephrased data while accuracy still improves, the proposed mechanism is not what drives the gains. A second decisive check is whether first-order fine-tuning improves as much as MeZO on the same rephrased data, which would indicate generic augmentation rather than optimization-aware rephrasing.

Watch

Extended reading notes

Core claim

The central claim is that OAT-Rephrase consistently improves MeZO-based fine-tuning across nearly all tasks and model scales, often narrowing or eliminating the gap with first-order methods. The paper attributes this to the reduction of unnecessary variance in loss values: training data with noisy or ambiguous phrasing makes MeZO's random-direction gradient estimates less stable, while semantically faithful rephrasing yields a smoother, more optimizable loss surface. In paired experiments sharing the same data indices, random seed, and MeZO noise vector, average accuracy gains of 2.28, 0.60, and 2.32 points are reported for Llama-3.2-1B, Llama-3.2-3B, and Mistral-7B-v0.1, while first-order baselines typically show flat or negative changes. This is presented as a reusable, low-overhead enhancement for zeroth-order tuning regimes.

Load-bearing premise

The load-bearing premise is that LLM rephrasing actually reduces the variance in MeZO's loss estimates and smooths the loss surface; the paper does not measure these quantities and only reports final accuracy.

Editorial extensions

If this is right

  • MeZO fine-tuned on OAT-Rephrase data outperforms MeZO on original data in 14 of 15 model-task pairs, with only RTE on Llama-3.2-3B declining.
  • The rephrased corpora are generated once per task and reused across model sizes, so the cost is a one-time preprocessing step rather than a per-model cost.
  • In several settings, MeZO with rephrased data matches or beats first-order full fine-tuning and LoRA trained on original data, narrowing the historical performance gap.
  • First-order fine-tuning and LoRA generally do not benefit from the rephrased data, which the paper takes as evidence the improvement is specific to zeroth-order dynamics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never directly measures the loss-landscape variance or gradient-estimate noise it invokes; a natural extension is to compute MeZO gradient-estimate variance on original versus rephrased data to test the proposed mechanism.
  • A stronger check on ZO-specificity would hold the rewrite instructions fixed and vary only whether the MeZO paper is included in the prompt; if gains persist without the paper, the optimization-awareness claim weakens.
  • The pipeline could be ported to other zeroth-order estimators, such as SPSA or random-subspace methods, or paired with sparse ZO, to see whether the rephrasing benefit transfers beyond MeZO.
  • Because all paired runs share a single random seed, a useful extension would repeat the comparison across several seeds and with different rewriter/judge models to establish robustness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces OAT-Rephrase, a data-centric preprocessing method for zeroth-order (MeZO) fine-tuning of LLMs. A rewriter LLM (GPT-4o) is prompted with content from the MeZO paper, task instructions, few-shot examples, and original training instances; an LLM-based judge then filters rewrites that do not preserve semantic and logical structure. The rephrased corpora are used to fine-tune Llama-3.2-1B/3B and Mistral-7B-v0.1 on COPA, CB, RTE, BoolQ, and ARC-C, under MeZO, full fine-tuning, and LoRA, comparing original versus rephrased training data. The paper reports average MeZO test-accuracy gains of 2.28, 0.60, and 2.32 points across the three backbones, and claims that OAT-Rephrase narrows or closes the gap to first-order methods.

Significance. The central claim is potentially valuable: a one-time, model-agnostic rephrasing of training data that improves MeZO without extra training cost would be a useful complement to algorithmic zeroth-order improvements. The experimental design has genuine strengths: paired runs share the same data indices, random seed, and optimizer noise; evaluation is on held-out test accuracy; first-order baselines on the same rephrased data provide a partial control for generic augmentation effects; and the rephrased datasets are generated once and reused across three backbones. However, the paper does not isolate the 'optimization-aware' mechanism it names, and the statistical basis for the main claim is thin: there is a single seed, no error bars, and no significance testing. The contribution is therefore not yet established at the level claimed in the abstract and Section 4.2.

major comments (4)
  1. [3.1, Phases 1-5 and 2.2] The causal claim that rephrasing 'based on its understanding of the ZO dynamics' improves MeZO is not actually tested. The rewriter prompt always contains the MeZO abstract, introduction, full-paper analysis, and algorithm summaries; there is no control condition that replaces this content with a generic paraphrase instruction, an unrelated paper, or a non-ZO optimizer description. The paper also never measures the hypothesized mechanism (loss-surface smoothness or gradient-estimate variance); only final test accuracy is reported. Without these controls, the observed gains could be produced by any high-quality paraphrase, and the 'optimization-aware' label is not supported. I would like to see an ablation that removes the MeZO content from the rewriter prompt and, if feasible, a direct measurement of directional-estimate variance on original versus rephrased data.
  2. [3.2 and Phase 3] The semantic-preservation guarantee is too weak to rule out label leakage. The judge is an LLM whose template is accepted at 90% agreement with human labels on a 40-pair test set, and the final corpora are not human-validated. The paper itself states in Phase 3 that the rewriter 'frequently' includes the correct answer in rewritten sentences; the prompt is revised to address this, but no estimate is given of leakage in the accepted corpus. Since leaked labels would give MeZO lexical shortcuts on test data, the Table 2 gains could be spurious. I recommend a human audit of a random sample of accepted rewrites per dataset, with the leakage rate reported, and an analysis of whether the rephrased data contains answer-revealing tokens.
  3. [4.1 and Table 2] All runs use a single seed (seed 0), and hyperparameters are chosen by maximizing development accuracy over a multi-dimensional grid (learning rate, batch size, LoRA rank, LoRA learning rate), with test results reported for the selected configuration. This procedure can inflate test accuracy through selection, and with one seed there is no estimate of stochastic variability. Because MeZO updates are noisy and several datasets are small, multiple seeds or significance testing are necessary to establish that the average gains (for example, 0.60 points on Llama-3.2-3B) are not noise. The paired design is a strength, but it does not remove the need for repeated runs.
  4. [4.2 and Table 2] The claim of 'consistent improvement' is overstated relative to the table. On Llama-3.2-3B, MeZO test accuracy on RTE drops from 66.4 to 61.7, a 4.7-point regression, and the average gain is only 0.60. Moreover, on the same rephrased data, FO-Full and FO-LoRA often degrade (for example, FO-Full average -2.14 and FO-LoRA -4.00 on Llama-3.2-3B), which weakens the interpretation that OAT-Rephrase is a general data-quality improvement and underscores the need for the mechanism analysis raised in the first major comment. Please report per-task variability and discuss these regressions explicitly.
minor comments (7)
  1. [5.1] The section heading 'Zeorth-Order LLM Fine-Tuning' contains a typo; it should read 'Zeroth-Order LLM Fine-Tuning'.
  2. [5.2] The sentence beginning 'n Recent advances...' has a stray leading 'n' and should be corrected to 'Recent advances...'.
  3. [4, RQ2] RQ2 refers to 'first-order optimization methods such as OAT-Rephrase'; OAT-Rephrase is a data rephrasing pipeline, not a first-order optimization method. The intended comparators are FO-Full and FO-LoRA.
  4. [Table 2] There are formatting inconsistencies in the table, such as '71.80' for Llama-3.2-3B FO-LoRA RTE, which has an extra zero, and inconsistent decimal formatting across entries.
  5. [Introduction and Section 2] There are spacing errors in the text, such as 'Tothisend' and 'Torepresent', which should be corrected to 'To this end' and 'To represent'.
  6. [3.1, Phase 2] The phrase 'based on the its understanding' has a grammatical error; it should be 'based on its understanding'.
  7. [Figure 2 caption] The caption uses '20 Original Data instances' with inconsistent capitalization; using sentence case consistently would improve readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central test-accuracy result is external and measured; the only minor self-citation is not load-bearing.

full rationale

The paper's central claim is empirical: OAT-Rephrase data, produced by a rewriter prompted with the MeZO paper, yields higher held-out test accuracy after MeZO fine-tuning than the original data (Table 2, Section 4.2). The improvement is measured on an external test set after training, so it is not an input to the pipeline. The rewriter prompt contains the MeZO paper and an explicit goal of making data 'better suited for training with MeZO' (Section 3.1, Phase 2), but this is an instruction to the LLM, not an assumption of success; the experiment can fail, and the first-order baselines (FO-Full, FO-LoRA) mostly do not improve, partially controlling for generic augmentation. The judge's 90% semantic-consistency gate (Section 3.2) is a quality filter for the generated corpus, not a definition of downstream performance. No equation is equivalent to another by construction: Eq. 1 defines the ZO estimator, and test accuracy in Table 2 is the outcome rather than a fitted quantity. The only self-citation bearing on experimental setup is Ref. [9] (Guo et al., ICLR 2025, with overlapping authorship), used for the train/dev/test split and cited in related work; this is minor and not load-bearing for the central claim. Residual concerns noted in the manuscript -- the 'optimization-aware' mechanism is not ablated, Phase 3 (Section 3.1) documents that the rewriter can inadvertently leak the correct answer into rewrites, and all runs use seed 0 with dev-selected configurations -- are threats to validity and mechanism confirmation, not circularity. The overall score of 2 reflects only the presence of the minor non-load-bearing self-citation; there is no reduction of the derivation to its inputs.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The experimental result depends on a set of swept hyperparameters whose chosen values are not reported, hand-chosen thresholds (δ, judge accuracy), and the unmeasured assumption that rephrasing changes the ZO loss landscape. No genuinely new entities such as particles or forces are postulated.

free parameters (7)
  • Zeroth-order perturbation magnitude δ = 1e-3
    Fixed for all MeZO runs; taken from standard MeZO settings rather than fit to these tasks, but it directly controls gradient estimate quality (Eq. 1).
  • Learning rate (MeZO and full fine-tuning) = not reported per run
    Swept over {1e-6, 5e-7, 2e-7, 1e-7}; the final value is chosen by dev accuracy in Section 4.1, a model-selection free parameter.
  • Batch size = not reported per run
    Swept over {2,4,8,16} with memory-adjusted alternatives; selected by dev accuracy.
  • LoRA rank = not reported
    Swept over {8,16,32}; selected by dev accuracy for LoRA baselines.
  • LoRA learning rate = not reported
    Swept over {1e-4, 2e-4}; selected by dev accuracy.
  • Judge accuracy threshold = 90%
    Hand-chosen acceptance criterion for the judge prompt template; controls how many rewrites survive into the training corpus.
  • Few-shot module size and judge test set size = 20 examples / 40 pairs
    Hand-picked; the judge template is iterated until it reaches 90% on this small 40-pair set per dataset.
assumptions (3)
  • domain assumption The ZO gradient estimator formula (Eq. 1) is a usable gradient proxy for LLM fine-tuning
    The whole method assumes MeZO is a meaningful optimization procedure worth improving; this is taken from the cited MeZO paper.
  • domain assumption Rephrasing training inputs while preserving semantics leaves the task and label distribution unchanged
    Assumed by design; enforced by an LLM judge whose accuracy is measured on only 40 pairs per dataset, not on the full corpus.
  • ad hoc to paper The rewriter LLM's internal understanding of MeZO dynamics, derived by reading the paper, translates into data that improves MeZO
    This is the paper's core hypothesis, stated in Section 2.2; no mechanism is proven, and the claimed smoother loss surface is not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OAT-Rephrase: Optimization-Aware Training Data Rephrasing for Zeroth-Order LLM Fine-Tuning." pith.science (2026). https://pith.science/paper/J5KGVSYF

@misc{pith2026250617264,
  author       = {Pith},
  title        = {Pith review of: OAT-Rephrase: Optimization-Aware Training Data Rephrasing for Zeroth-Order LLM Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5KGVSYF}},
  note         = {Machine review of arXiv:2506.17264}
}
read the original abstract

Fine-tuning large language models (LLMs) using zeroth-order optimization (ZO) offers a memory-efficient alternative to gradient-based methods but suffers from slower convergence and unstable optimization due to noisy gradient estimates. This paper introduces OAT-Rephrase, an Optimization-Aware Training data rephrasing strategy that leverages an LLM to rephrase training instances based on its understanding of the ZO dynamics, specifically MeZO, derived directly from its paper. The approach incorporates a dual-stage pipeline featuring a rewriter LLM and a semantic judge, ensuring all rephrasings retain task relevance and logical consistency. Evaluations across five classification tasks and three LLM architectures demonstrate that OAT-Rephrase consistently improves MeZO fine-tuning performance, often narrowing or eliminating the gap with first-order methods. Our findings suggest that optimization-aware rephrasing serves as a reusable and low-overhead enhancement for zeroth-order tuning regimes.

Figures

Figures reproduced from arXiv: 2506.17264 by the authors.

Figure 1
Figure 1. OAT-Rephrase workflow. (i) A Rewriter Prompt Template encodes task instructions, key requirements, and few-shot examples. Populated with an original instance (shown in red), it forms a Rewriter Prompt. (ii) An LLM generates a rewritten version (shown in blue). (iii) The original and rewritten content are composed into a Judge Prompt via a Judge Prompt Template. (iv) An LLM verifies whether the rewritten version pres… view at source ↗
Figure 2
Figure 2. Construction of the Few-Shot In￾struction Module. (i) 20 Original Data instances are inserted into the Phase-4 Prompt Template to generate zero-shot rewriter prompts, with the rewrite span marked in red. (ii) The LLM rewrites each span, yielding OAT-Rephrase outputs with rewritten text shown in blue. (iii) All rewrit￾ten results are manually inspected and paired with their original instance to form the Few￾Shot Inst… view at source ↗
Figure 3
Figure 3. The Judge Prompt Template starts with task instructions and five few￾shot examples. In each refinement cycle, an Original-Synthetic Pair is added to form a Judge Prompt: spans copied from the original are shown in red, their rewrites in blue. The LLM evaluates forty prompts, and the resulting judge accuracy, defined as the proportion of model decisions that match hu￾man judgments, determines the process. A template … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Rewriter Prompt Template after Phase 1. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Rewriter Prompt Template after Phase 2. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Rewriter Prompt Template after Phase 3. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Rewriter Prompt Template after Phase 4. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Rewriter Prompt Template after Phase 5. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Judge Prompt Template Initialization. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: The Final Version of the Judge Prompt Template. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 15 canonical work pages

  1. [1]

    Languagemodels are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, ArvindNeelakantan, PranavShyam, GirishSastry, AmandaAskell, etal. Languagemodels are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  2. [2]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Mark Gardner, and Others. Boolq: Exploring the surprising difficulty of natural yes/no questions. InProceedings of NAACL-HLT, 2019. 10

    work page 2019

  3. [3]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. InarXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  4. [4]

    The commitmentbank: Investigating projection in naturally occurring discourse

    Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. InProceedings of Sinn und Bedeutung, 2019

    work page 2019

  5. [5]

    Data augmentation using llms: Data per- spectives, learning paradigms and challenges

    Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Luu Anh Tuan, and Shafiq Joty. Data augmentation using llms: Data per- spectives, learning paradigms and challenges. InFindings of the Association for Computational Linguistics ACL 2024, pages 1679–1705, 2024

    work page 2024

  6. [6]

    Parameter-efficient fine-tuning of large-scale pre-trained language models.Nature Machine Intelligence, 5(3):220–235, 2023

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models.Nature Machine Intelligence, 5(3):220–235, 2023

    2023

  7. [7]

    Variance-reduced zeroth-order methods for fine-tuning language models

    Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models. InForty-first In- ternational Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URLhttps://openreview.net/forum?id=VHO4nE7v41

    work page 2024

  8. [8]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 24 references
  1. [9]

    Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhuo Xu

    Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhuo Xu. Zeroth-order fine-tuning of llms with transferable static sparsity. InThe Thirteenth International Conference on...

    2025

  2. [10]

    Generate, annotate, andlearn: Nlpwithsynthetictext.Transactions of the Association for Computational Linguistics, 10:826–842, 2022

    XuanliHe, IslamNassar, JamieKiros, GholamrezaHaffari, andMohammadNorouzi. Generate, annotate, andlearn: Nlpwithsynthetictext.Transactions of the Association for Computational Linguistics, 10:826–842, 2022

    2022

  3. [11]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Ope...

    2022

  4. [12]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  5. [13]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thoma...

    doi arXiv

  6. [14]

    Zeroth-order stochastic variance reduction for nonconvex optimization.Advances in neural information processing systems, 31, 2018

    Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. Zeroth-order stochastic variance reduction for nonconvex optimization.Advances in neural information processing systems, 31, 2018

    2018

  7. [15]

    Black, Adrian Weller, and Bernhard Schölkopf

    Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. InThe Twelfth Int...

    2024

  8. [16]

    Winner- take-all column row sampling for memory efficient adaptation of language model.Advances in Neural Information Processing Systems, 36:3402–3424, 2023

    Zirui Liu, Guanchu Wang, Shaochen Henry Zhong, Zhaozhuo Xu, Daochen Zha, Ruixiang Ryan Tang, Zhimeng Stephen Jiang, Kaixiong Zhou, Vipin Chaudhary, Shuai Xu, et al. Winner- take-all column row sampling for memory efficient adaptation of language model.Advances in Neural Inform...

    2023

  9. [17]

    On LLMs-driven synthetic data generation, curation, and evaluation: A survey

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On LLMs-driven synthetic data generation, curation, and evaluation: A survey. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Com- putational Linguist...

    2024 doi

  10. [18]

    Fine-tuning language models with just forward passes.Advances in Neural Information Processing Systems, 36:53038–53075, 2023

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes.Advances in Neural Information Processing Systems, 36:53038–53075, 2023

    2023

  11. [19]

    Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. InCommonsense 2011, 2011

    2011

  12. [20]

    In defense of structural sparse adapters for concurrent llm serving

    Junda Su, Zirui Liu, Zeju Qiu, Weiyang Liu, and Zhaozhuo Xu. In defense of structural sparse adapters for concurrent llm serving. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 4948–4953, 2024

    2024

  13. [21]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP, 2018

    2018

  14. [22]

    Subzero: Random subspace zeroth- order optimization for memory-efficient llm fine-tuning.arXiv preprint arXiv:2410.08989, 2024

    Ziming Yu, Pan Zhou, Sike Wang, Jia Li, and Hua Huang. Subzero: Random subspace zeroth- order optimization for memory-efficient llm fine-tuning.arXiv preprint arXiv:2410.08989, 2024

    2024 arXiv

  15. [23]

    Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen

    Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Forty-fi...

    2024

  16. [24]

    A survey on data augmentation in large model era.arXiv preprint arXiv:2401.15422, 2024

    Yue Zhou, Chenlu Guo, Xu Wang, Yi Chang, and Yuan Wu. A survey on data augmentation in large model era.arXiv preprint arXiv:2401.15422, 2024. 12 Appendix A Prompts for OAT-Rephrase We present the prompts in each phase of OAT-Rephrase for COPA dataset as an illustration. 13 Fig...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.