Pith. sign in

REVIEW 3 major objections 6 minor 69 references

Exploring Efficient Learning of Small BERT Networks with LoRA and DoRA

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a small BERT model needs only rank-1 low-rank weight updates: LoRA at rank 1 matches higher-rank and full fine-tuning, and automatic mixed precision adds memory and time savings with negligible accuracy loss.

desk verdict Solid efficiency measurements on a small BERT are undermined by an overclaimed 'intrinsic rank is 1' conclusion that the experiments do not support. read the letter →

arxiv 2508.17586 v1 pith:7DPWI2I3 submitted 2025-08-25 cs.LG

Daniel Frees , Aditri Bhagirath , Moritz Bolling This is my paper · ORCID
classification cs.LG
keywords LoRADolow-rankadaptationparameter-efficientfine-tuningminBERTautomaticmixedprecisionintrinsicdimensionmulti-tasklearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the efficiency techniques developed for very large language models—low-rank adaptation (LoRA and its weight-decomposed variant DoRA) and automatic mixed precision—still pay off when the model being fine-tuned is small. Using the compact minBERT architecture, a small BERT variant, on sentiment, paraphrase detection, and semantic similarity tasks, it reports that they do: memory use falls by up to 50% and training time by up to 60% without a consistent accuracy penalty. The sharper claim is about why: fine-tuning updates to this small BERT have intrinsic rank 1, so a rank-1 LoRA decomposition captures the update signal, and pushing the rank higher does not help. The paper also finds that DoRA's extra magnitude-and-direction decomposition buys nothing at this scale, and that the compute freed up by efficient fine-tuning is best spent ensembling many models, which produces its strongest multitask scores.

What carries the argument

The load-bearing objects are the low-rank decompositions themselves. LoRA freezes the pretrained weight matrix $W_0$ and represents the update as $W' = W_0 + BA$, with trainable matrices $B$ and $A$ joined along a shared rank $r$; DoRA refines this by separating magnitude from direction, $W' = m \cdot (W_0 + BA)/\|W_0 + BA\|$, with $m$ a trainable magnitude vector. The intrinsic-rank-1 argument is carried by sweeping $r$ from 1 to 100 and observing that accuracy is flat in $r$ while memory and computation rise. Automatic mixed precision—forward passes and losses in float16, gradients accumulated in float32 with loss scaling, and master weights kept in float32—supplies the efficiency half of the claim. Four LoRA/DoRA application modes (attention-only or all linear layers, with either frozen or trainable backbones) plus an ablation against full fine-tuning connect the decomposition identity to the measured memory, time, and accuracy numbers.

What would settle it

Run the identical LoRA rank sweep (ranks 1, 2, 5, 10, 20, 50, 80, 100) on the Quora paraphrase task, which the paper excluded for compute reasons, and on a second small BERT variant with different depth or width; the intrinsic-rank-1 claim fails if accuracy rises measurably with rank on either, or if rank 1 falls more than a few points short of the best higher rank. A cheaper check is recomputing the per-task normalized accuracy gap between rank 1 and the best higher rank from the paper's own rank-sweep figure.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the low intrinsic dimensionality observed when fine-tuning large language models persists at the small end of the spectrum. In grid searches spanning LoRA and DoRA across ranks 1 through 100 on minBERT, normalized accuracy stayed flat once the rank reached 1, while memory and time grew with rank, which the authors take as evidence that the weight update $\Delta W$ during small-BERT fine-tuning has intrinsic dimension 1 on these tasks. On the efficiency side, combining LoRA applied to all linear transformer layers with automatic mixed precision reduces peak memory by up to 50% and per-epoch time by up to 60% relative to full fine-tuning, and in the final-model comparison LoRA plus AMP slightly outperformed the full fine-tuning baseline while DoRA plus AMP stayed within a one-percent band. The paper also reports task-dependent sensitivity, with STS similarity losing about 4% more accuracy than sentiment analysis under low-rank adaptation, which it reads as evidence that the intrinsic dimension of updates varies by task.

Load-bearing premise

The rank-1 conclusion rests on rank sweeps run on only two of the three tasks (SST and STS) and on a single minBERT variant; if those are not representative of the paraphrase task or of small BERT networks generally, the claim that fine-tuning updates have intrinsic rank 1 would not hold broadly.

Editorial extensions

If this is right

  • Practitioners fine-tuning small BERT-scale models can set the LoRA rank to 1 and expect no meaningful accuracy loss, pocketing the maximum parameter and memory savings.
  • AMP combined with LoRA on all linear transformer layers is the recommended efficiency configuration, delivering up to 50% memory reduction and 60% time speedups with accuracy within noise of full fine-tuning.
  • Because individual fine-tuning runs become cheap, the bottleneck shifts to other resources: the authors reinvest the savings into hyperparameter search and ensembling, and their 23-model ensemble reaches an overall multitask test score of 0.788.
  • DoRA's magnitude-direction decomposition is not needed at small scale; plain LoRA matches or exceeds it, so the simpler method is the better default.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the intrinsic-rank-1 result generalizes, the default configuration for fine-tuning compact encoders should become rank-1 LoRA on all linear layers plus mixed precision, with the relative benefit largest for small models because the savings are biggest compared with their total parameter count.
  • The paper's own observation that STS loses about 4% more accuracy than SST under low-rank adaptation suggests intrinsic dimension is task-dependent; a direct test would be a per-task rank sweep measuring where accuracy saturates for each task separately.
  • The rank sweeps ran on one minBERT variant and on only two of the three tasks because Quora was too compute-intensive, so whether rank-1 sufficiency extends to the paraphrase task and to other small BERT architectures (different depth, width, or distillation) is untested.
  • A strategic corollary of the ensemble results: when fine-tuning is cheap, optimal compute allocation shifts from training one large model to ensembling many cheap rank-1 models, a pattern likely to dominate small-model leaderboards.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports an empirical study of LoRA and DoRA applied to a compact minBERT model (roughly 110M parameters per backbone, about 226M total in the final dual-backbone configuration) for three tasks: SST sentiment analysis, Quora paraphrase detection, and STS semantic textual similarity. The authors implement LoRA and DoRA from scratch, benchmark them under several modes with and without AMP, and report memory savings of 7% to 50% and time speedups of up to 60% with generally small accuracy changes, including a three-seed final comparison. They then use the efficient training setup to explore optimizers, loss functions, unfreezing schedules, and ensembling, ultimately producing a 23-member ensemble. The paper also claims in Section 6 that fine-tuning updates to small BERT models have intrinsic rank 1, based on rank-sweep experiments in Appendix M.2.

Significance. If the efficiency numbers are accurate, the paper provides a useful practical benchmark of LoRA, DoRA, and AMP for small BERT models on a single GPU, which is a genuinely underserved setting in the parameter-efficient fine-tuning literature. Strengths include the from-scratch custom implementations, the detailed hyperparameter tables, the three-seed final comparison, and the honest disclosure of compute-related limitations in Appendix M.2. The observation that rank-1 LoRA is sufficient on the evaluated tasks is interesting, as is the finding that DoRA does not outperform LoRA on minBERT. However, the central conceptual conclusion that the intrinsic dimension of the weight updates is 1 is not supported by the evidence presented, and the paper itself contains a statement in Section 5 that suggests task-specific intrinsic dimensions. The practical efficiency contribution can stand after revision, but the intrinsic-dimension claim must be either removed or substantially qualified.

major comments (3)
  1. [Section 6 / Appendix M.2] The claim in Section 6 that "the intrinsic dimension of the weight updates, ΔW, is 1" is not established by the experiments. The rank sweep in Appendix M.2 was run only on SST and STS, on a single minBERT variant, because the Quora task was excluded for compute reasons, and no confidence intervals or multiple-seed statistics are reported for that sweep. More fundamentally, showing that rank-1 LoRA matches higher-rank LoRA on two tasks does not measure the rank of the ΔW produced by full fine-tuning; a rank-1 LoRA update still updates many parameters per adapted matrix and constrains only the row/column space of that matrix. In the intrinsic-dimensionality literature (Li et al. 2018; Aghajanyan et al. 2020), intrinsic dimension is the minimum dimension of a parameter subspace that recovers full-training performance, not the rank of a per-layer update matrix. I recommend replacing the Section 6 claim with a statement about rank-1 LoRA/DoRA sufficiency on the evaluated tasks and reporting per-task, per-rank error bars.
  2. [Section 5, first paragraph] The paper's own analysis contradicts the universal rank-1 claim. Section 5 states that "STS tasks suffer an average performance decrease of 4% more compared to sentiment classification tasks, suggesting task-specific changes in the intrinsic dimension of the weight updates." If the intrinsic dimension varies by task, it cannot be uniformly 1 for all three tasks and for small BERT networks generally. The conclusion in Section 6 should be restricted to the tasks and architecture actually evaluated, and the relationship between the Section 5 task-specific observation and the Section 6 universal claim needs to be reconciled explicitly.
  3. [Appendix M.2 / Figure 20] The conclusion in Section 4.4.1 that "accuracy does not improve with increasing rank" is not verifiable from the reported data. Figure 20 presents normalized memory, time, and accuracy values but includes no error bars, no per-seed scatter, and no statement about the number of replicates per condition. Without this information, the flat accuracy-versus-rank pattern could be within run-to-run noise, especially since the final three-seed comparison elsewhere in the paper reports variance within 1%. Please report means and standard deviations over at least three seeds for the rank sweep, or explicitly label the rank sweep as exploratory and remove the load-bearing inference drawn from it.
minor comments (6)
  1. [Throughout] The manuscript contains many encoding artifacts and garbled equations (for example, §2.2, §3.4.4, and Appendix F), and the LoRA mode names in §3.4.5 are not rendered correctly; these need to be fixed for reproducibility and readability.
  2. [Figure 2 / Figure 20] Figure 2 says values are "Normalized to Full Model baseline," while Figure 20 says values are "Normalized to LoRA and Rank 1"; please clarify the normalization for each figure and state how accuracy is aggregated across tasks.
  3. [Appendix M.2, first paragraph] The numbers 960, 416, and 270 for grid-search combinations are not fully explained; please enumerate the infeasible configurations and state explicitly how many completed runs contribute to each reported point.
  4. [Appendix F / Section 3.3] Appendix F states that a mixed BCE/MSE loss led to the best SST performance, but Section 3.3 says the final model uses cross-entropy loss for SST; this discrepancy should be resolved.
  5. [Table 2 / Section 4.3] Please state whether the test-set results in Table 2 come from a single ensemble evaluation or multiple seeds, and report the number of models in each ensemble in the table caption.
  6. [References / Section 7] Several references are malformed or incomplete (for example, the Hu et al. citation in the LoRA section and the in-text citation "dwt (2024)" in the ethics statement); these should be corrected before publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the LoRA/DoRA efficiency benchmarks are self-contained, and the 'intrinsic rank 1' statement is an overbroad empirical inference rather than an input-equivalent prediction.

full rationale

The paper does not fit a parameter and then relabel the fit as a prediction, nor does it rely on a self-citation chain or an author-imported uniqueness theorem. Its efficiency and accuracy numbers are measured against its own baselines (full fine-tuning without AMP, and a classifier-only lower bound) in Section 4.4 and Appendices M.2/M.3, so those results are self-contained. The only sentence that could look circular is Section 6's inference that rank-1 LoRA/DoRA matching higher ranks means 'the intrinsic dimension of the weight updates, ΔW, is 1'; however, the paper does not define intrinsic dimension as LoRA rank or derive the claim by construction. It is an unsupported generalization from rank sweeps restricted to SST/STS on one architecture, and Section 5 itself notes task-specific behavior ('STS tasks suffer an average performance decrease of 4% more compared to sentiment classification tasks, suggesting task-specific changes in the intrinsic dimension of the weight updates'). That is an evidential and scope gap, not a circular reduction, so no circularity step is scored.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claims rest mainly on hyperparameters chosen by human experimentation and on the assumption that LoRA and DoRA parameterizations faithfully expose the intrinsic rank of fine-tuning updates. No new scientific entities are introduced.

free parameters (6)
  • Task-specific learning rate multipliers = SST 4x, Paraphrase 5x, STS 3x
    Chosen by hand after experiments in Appendices G and I; the final 'optimal' model depends on these values.
  • Learning rate decay lambda = 0.5
    Final model uses multiplicative decay 0.5 per epoch (Table 7); selected by experiments shown in Figure 19.
  • Per-task weight decays = SST 9e-3, Paraphrase 1e-5, STS 1e-2
    Final model hyperparameters in Table 7; hand-tuned during the optimization search.
  • LoRA rank = 1
    Rank experiments in M.2 show no accuracy gain with rank, and rank 1 is used in the final efficiency benchmark. This value underpins the paper's 'intrinsic dimension is 1' claim.
  • Backbone unfreezing schedule = 3 attention layers per epoch, embeddings at final unfreeze
    Iterative unfreezing schedule chosen in Appendix B; it affects all final model results.
  • Ensemble composition and vote weights = 23 sub-models, unweighted mean
    Final 'Lisan al Gaib' ensemble includes one of almost everything; adding more models was found to help in Appendix L, but the exact composition is arbitrary.
assumptions (4)
  • domain assumption Pre-trained minBERT weights provide a suitable initialization for SST, QQP, and STS.
    The entire fine-tuning setup assumes transfer learning from the pre-trained backbone is valid for all three downstream tasks.
  • domain assumption The LoRA and DoRA decompositions can represent the fine-tuning updates that matter for downstream performance.
    Formulas in Sections 3.4.3 and 3.4.4 are used to measure efficiency and rank sensitivity; if this parameterization is misspecified, the rank-1 conclusion does not follow.
  • domain assumption The simple average of normalized task metrics is a meaningful multi-task objective.
    Used for checkpointing and final scores in Section 4.2, following the CS224N leaderboard formula.
  • domain assumption Training results on SST and STS generalize to Quora for rank conclusions.
    Appendix M.2 restricts the rank sweep to SST and STS due to compute, yet Section 6 concludes about weight updates across the multi-task model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Efficient Learning of Small BERT Networks with LoRA and DoRA." pith.science (2026). https://pith.science/paper/7DPWI2I3

@misc{pith2026250817586,
  author       = {Pith},
  title        = {Pith review of: Exploring Efficient Learning of Small BERT Networks with LoRA and DoRA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7DPWI2I3}},
  note         = {Machine review of arXiv:2508.17586}
}
read the original abstract

While Large Language Models (LLMs) have revolutionized artificial intelligence, fine-tuning LLMs is extraordinarily computationally expensive, preventing smaller businesses and research teams with limited GPU resources from engaging with new research. Hu et al and Liu et al introduce Low-Rank Adaptation (LoRA) and Weight-Decomposed Low-Rank Adaptation (DoRA) as highly efficient and performant solutions to the computational challenges of LLM fine-tuning, demonstrating huge speedups and memory usage savings for models such as GPT-3 and RoBERTa. We seek to expand upon the original LoRA and DoRA papers by benchmarking efficiency and performance of LoRA and DoRA when applied to a much smaller scale of language model: our case study here is the compact minBERT model. Our findings reveal that optimal custom configurations of LoRA and DoRA, coupled with Automatic Mixed Precision (AMP), significantly enhance training efficiency without compromising performance. Furthermore, while the parameterization of minBERT is significantly smaller than GPT-3, our results validate the observation that gradient updates to language models are inherently low-rank even in small model space, observing that rank 1 decompositions yield negligible performance deficits. Furthermore, aided by our highly efficient minBERT implementation, we investigate numerous architectures, custom loss functions, and hyperparameters to ultimately train an optimal ensembled multitask minBERT model to simultaneously perform sentiment analysis, paraphrase detection, and similarity scoring.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 68 canonical work pages

  1. [1]

    Dev sentiment accuracy: 0.520

    work page

  2. [3]

    Duality of Man

    Dev STS correlation: 0.611 Figure 7: Adamax (dark blue) yields smoother performance increase and better convergence as opposed to Adam and SGD (grey). Adamax + Linear Similarity Feature Extraction layer between minBERT and Quora/STS tasks further improves performance (light blue) 24 We then implemented a number custom loss functions, including a Pearson C...

    work page

  3. [4]

    Oligarchy

    Overall test score: 0.786 G6 "Oligarchy" v2 This ensemble is the same as Oligarchy v1 except we give Yin Yang, Yang Yin, Yang Yin the Parabola, and Duality of Robust Men models additional votes as well. $python ensemble.py –filepaths g6-yin-yang.pt g6-yang-yin.pt g6-yin-yang-fast.pt g6-yin-yang-steady-hand.pt g6-yang-yin-the-parabola.pt g6-yin-yang-2.pt g...

    work page

  4. [8]

    Dev sentiment accuracy: 0.396

    work page

  5. [10]

    We then upgraded the Similarity Feature Extraction layer to be non-linear instead of linear and noted further STS performance improvements, as shown in Figure 9

    Dev STS correlation: 0.688 Figure 8: Effect of Pearson Correlation Loss Optimizer (Purple) on STS and SST performance. We then upgraded the Similarity Feature Extraction layer to be non-linear instead of linear and noted further STS performance improvements, as shown in Figure 9. Result (no Quora training):

    work page

  6. [11]

    Dev sentiment accuracy: 0.384

    work page

  7. [12]

    Dev paraphrase accuracy: 0.632

    work page

  8. [13]

    Sent" (SST) and

    Dev STS correlation: 0.737 Figure 9: NonLinear Similarity Feature Layer improves STS (light green). 25 Towards gaining a rough-and-dirty approximation of Quora performance, we then implemented partial Quora training whereby we would randomly select approximately 5% (200 batches at batch size 96) ofQuora data to get a rough and dirty train and benchmark. U...

    work page

Show all 69 references
  1. [14]

    Dev sentiment accuracy: 0.510

  2. [15]

    Dev paraphrase accuracy: 0.368

  3. [16]

    Dev STS correlation: 0.760 Figure 14: Effect of separating minBERT backbones (orange) and introducingnltk preprocessing (purple) on performance. We ran a massive 50 epoch (10 epoch cycles for each unfreezing of1/4 of the backbone) experiment for 35 epochs overnight, and notice...

  4. [17]

    Sentiment classification accuracy: 0.487

  5. [18]

    Paraphrase detection accuracy: 0.368

  6. [19]

    Duality of Man

    Semantic Textual Similarity correlation: 0.651 Figure 16: Impact of lower learning rate and lambda = 0.7**epoch scheduler on performance (purple). We then tried a triangular cyclical learning rate schedule from 1e � 4 to 2e � 4, seeing very little difference. With the cylical ...

  7. [20]

    Sentiment classification accuracy: 0.533

  8. [21]

    Paraphrase detection accuracy: 0.858

  9. [22]

    Yin Yang

    Semantic Textual Similarity correlation: 0.853 32 I "Yin Yang" Model Variants and Ensemble Towards improving the "Duality of Man" variants architecture, the Yin Yang model architecture (our best multi-task model architecture) includes both pure sentence embeddings and the extr...

  10. [23]

    Dev sentiment accuracy: 0.516

  11. [24]

    Dev paraphrase accuracy: 0.851

  12. [25]

    Yang Yin

    Dev STS correlation: 0.861 I.2 G6 "Yang Yin" $python multitask_classifier.py –fine-tune-mode iterative –lr 1e-4 –use_gpu –amp –batch_size 64 –train_sst –train_quora –train_sts –clf conv –sst_weight_decay 8e-3 –para_weight_decay 1e-5 –sts_weight_decay 9e-3 –lr_lambda 0.55 –opti...

  13. [26]

    Dev sentiment accuracy: 0.515

  14. [27]

    Dev paraphrase accuracy: 0.888

  15. [28]

    Yin Yang Fast

    Dev STS correlation: 0.835 I.3 G6 "Yin Yang Fast" $python multitask_classifier.py –fine-tune-mode iterative –lr 1.5e-4 –use_gpu –amp –batch_size 64 –train_sst –train_quora –train_sts –clf conv –sst_weight_decay 8e-3 –para_weight_decay 1e-5 –sts_weight_decay 9e-3 34 –lr_lambda ...

  16. [29]

    Dev sentiment accuracy: 0.513

  17. [30]

    Dev paraphrase accuracy: 0.887

  18. [31]

    Yin Yang Steady Hand

    Dev STS correlation: 0.816 I.4 G6 "Yin Yang Steady Hand" $python multitask_classifier.py –fine-tune-mode iterative –lr 1e-4 –use_gpu –amp –batch_size 64 –train_sst –train_quora –train_sts –clf conv –sst_weight_decay 9e-3 –para_weight_decay 1e-5 –sts_weight_decay 1e-2 –lr_lambd...

  19. [34]

    Yang Yin The Parabola

    Semantic Textual Similarity correlation: 0.848 I.5 G6 "Yang Yin The Parabola" $python multitask_classifier.py –fine-tune-mode iterative –lr 1e-4 –use_gpu –amp –batch_size 64 –train_sst –train_quora –train_sts –clf conv –sst_weight_decay 9e-3 –para_weight_decay 1e-5 –sts_weight...

  20. [35]

    Sentiment classification accuracy: 0.515

  21. [36]

    Paraphrase detection accuracy: 0.892

  22. [37]

    Yin Yang

    Semantic Textual Similarity correlation: 0.801 I.6 G6 "Yin Yang" 2 Minor adjustments to the original to better time the SST optimal point towards the 7th epoch. $python multitask_classifier.py –fine-tune-mode iterative –lr 1e-4 –use_gpu –amp –batch_size 64 –train_sst –train_qu...

  23. [39]

    Paraphrase detection accuracy: 0.842

  24. [40]

    Yue the Moon Spirit

    Semantic Textual Similarity correlation: 0.858 I.7 G6 "Yue the Moon Spirit" - "Yin Yang" Ensemble We ensemble all of the Yin Yang variants (other than Steady Hand 2,3,4,5,6 which were not yet available) via: $python ensemble.py –filepaths g6-yin-yang.pt g6-yang-yin.pt g6-yin-y...

  25. [41]

    Sentiment classification accuracy: 0.522

  26. [43]

    Yin Yang Steady Hand

    Semantic Textual Similarity correlation: 0.858 Based on the success of the Steady Hand variant, we later made more to add to our final ensembles. I.8 G6 "Yin Yang Steady Hand" 2 $python multitask_classifier.py –fine-tune-mode iterative –lr 1e-4 –use_gpu –amp –batch_size 64 –tr...

  27. [44]

    Sentiment classification accuracy: 0.510

  28. [45]

    Paraphrase detection accuracy: 0.887

  29. [46]

    Yin Yang Steady Hand

    Semantic Textual Similarity correlation: 0.845 I.11 G6 "Yin Yang Steady Hand" 5 $python multitask_classifier.py –fine-tune-mode iterative –lr 1e-4 –use_gpu –amp –batch_size 64 –train_sst –train_quora –train_sts –clf conv –sst_weight_decay 9e-3 –para_weight_decay 1e-5 –sts_weig...

  30. [47]

    Sentiment classification accuracy: 0.520

  31. [48]

    Paraphrase detection accuracy: 0.888

  32. [49]

    Yin Yang Steady Hand

    Semantic Textual Similarity correlation: 0.841 I.12 G6 "Yin Yang Steady Hand" 6 $python multitask_classifier.py –fine-tune-mode iterative –lr 1e-4 –use_gpu –amp –batch_size 64 –train_sst –train_quora –train_sts –clf conv –sst_weight_decay 9e-3 –para_weight_decay 1e-5 –sts_weig...

  33. [50]

    Sentiment classification accuracy: 0.518

  34. [52]

    Run it Back

    Semantic Textual Similarity correlation: 0.840 37 J G6 "Run it Back" Model Variant Given the massively larger compute requirements of training full epochs on the Quora dataset, we proposed that we might get easy value with minimal added compute by re-shuffling and re-training ...

  35. [53]

    Dev sentiment accuracy: 0.494

  36. [54]

    Dev paraphrase accuracy: 0.653

  37. [55]

    Interleaver

    Dev STS correlation: 0.839 38 K G6 "Interleaver" Model Variants and Ensemble As opposed to the Duality of Man, Yin Yang, and Run it Back architectures, the Interleaver models derive their name from the fact that we interleave model training at a finer granularity. Whereas all ...

  38. [56]

    Sentiment classification accuracy: 0.500

  39. [57]

    Paraphrase detection accuracy: 0.897

  40. [58]

    Ménage à Onze

    Semantic Textual Similarity correlation: 0.866 40 L Final Ensembles L.1 G6 "Ménage à Onze" - Duality of Man + Yin Yang Ensemble We ensemble all Yin Yang and Duality of Man variants with: $python ensemble.py –filepaths g6-yin-yang.pt g6-yang-yin.pt g6-yin-yang-fast.pt g6-yin-ya...

  41. [59]

    Sentiment classification accuracy: 0.538

  42. [60]

    Paraphrase detection accuracy: 0.879

  43. [61]

    Oligarchy

    Semantic Textual Similarity correlation: 0.860 Other Ensembles Tried: G6 "Oligarchy" v1 [First CS224N Class Test Leaderboard Submission] This model is the same as G6 "Ménage à Onze" except that we now give Yin Yang Steady Hand (our most performant single sub-model) 3 votes. $p...

  44. [62]

    Sentiment classification accuracy: 0.536

  45. [63]

    Paraphrase detection accuracy: 0.882

  46. [64]

    Semantic Textual Similarity correlation: 0.860 Test Result:

  47. [65]

    SST test accuracy: 0.551

  48. [66]

    Paraphrase test accuracy: 0.881

  49. [67]

    STS test correlation: 0.853

  50. [70]

    Paraphrase detection accuracy: 0.883 41

  51. [71]

    Lisan al Gaib

    Semantic Textual Similarity correlation: 0.860 G6 "Lisan al Gaib" This ensemble includes one of (almost) everything, including additional slightly modified versions of Yin Yang Steady Hand. Kitchen-sink style ensemble. It sees across the dunes. $python ensemble.py –filepaths g...

  52. [72]

    Sentiment classification accuracy: 0.530

  53. [73]

    Paraphrase detection accuracy: 0.891

  54. [74]

    The Sage

    Semantic Textual Similarity correlation: 0.864 G6 "The Sage" This ensemble combines primarily known high-quality sub-models, with a handful of weaker per- formers. It does not perform better than the kitchen sink model, suggesting that simply including more models is more opti...

  55. [75]

    Sentiment classification accuracy: 0.529

  56. [76]

    Paraphrase detection accuracy: 0.889

  57. [77]

    n’t" with

    Semantic Textual Similarity correlation: 0.860 42 M Efficiency Experiments Details M.1 Grid Search Details Parameter Values AMP On Learning Rate 1 � 10� �, 5 � 10� �, 1 � 10� �, 5 � 10� � Batch Size 64, 128, 256, 384 LoRA Rank 1, 5, 10 LoRA Mode ’none’, ’all-lin’, ’attn’, ’all...

    2009

  58. [2015]

    Software available from tensorflow.org

    Tensorboard: Visualize your tensorflow model. Software available from tensorflow.org

  59. [2019]

    Quora Question Pairs

    Averaging weights leads to wider optima and better generalization. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ����� �������� �����...

    2015

  60. [2023]

    https://paperswithcode.com/method/polyak-averaging

    Polyak averaging. https://paperswithcode.com/method/polyak-averaging. Ac- cessed: 2024-06-08

    2024

  61. [2024]

    https://www.dwt.com/insights/ 2024/05/ai-bias-audits-using-ai-for-hiring-employment

    Ai bias audits: Using ai for hiring & employment. https://www.dwt.com/insights/ 2024/05/ai-bias-audits-using-ai-for-hiring-employment . Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimensionality explains the effectiveness of language model fine-tuning....

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.