Pith. sign in

REVIEW 9 cited by

Label Supervised LLaMA Finetuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01208 v1 pith:4DDIBNKL submitted 2023-10-02 cs.CL

Zongxi Li , Xianming Li , Yuzhang Liu , Haoran Xie , Jing Li , Fu-lee Wang , Qing Li , Xiaoqin Zhong This is my paper · ORCID
classification cs.CL
keywords llmslabeltasksapproachclassificationfinetuningllamasupervised
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The recent success of Large Language Models (LLMs) has gained significant attention in both academia and industry. Substantial efforts have been made to enhance the zero- and few-shot generalization capabilities of open-source LLMs through finetuning. Currently, the prevailing approach is instruction-tuning, which trains LLMs to complete real-world tasks by generating responses guided by natural language instructions. It is worth noticing that such an approach may underperform in sequence and token classification tasks. Unlike text generation tasks, classification tasks have a limited label space, where precise label prediction is more appreciated than generating diverse and human-like responses. Prior research has unveiled that instruction-tuned LLMs cannot outperform BERT, prompting us to explore the potential of leveraging latent representations from LLMs for supervised label prediction. In this paper, we introduce a label-supervised adaptation for LLMs, which aims to finetuning the model with discriminant labels. We evaluate this approach with Label Supervised LLaMA (LS-LLaMA), based on LLaMA-2-7B, a relatively small-scale LLM, and can be finetuned on a single GeForce RTX4090 GPU. We extract latent representations from the final LLaMA layer and project them into the label space to compute the cross-entropy loss. The model is finetuned by Low-Rank Adaptation (LoRA) to minimize this loss. Remarkably, without intricate prompt engineering or external knowledge, LS-LLaMA substantially outperforms LLMs ten times its size in scale and demonstrates consistent improvements compared to robust baselines like BERT-Large and RoBERTa-Large in text classification. Moreover, by removing the causal mask from decoders, LS-unLLaMA achieves the state-of-the-art performance in named entity recognition (NER). Our work will shed light on a novel approach to adapting LLMs for various downstream tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Alignment on Llama 3 reduces explicit bias but amplifies implicit bias, because aligned models no longer represent 'black' and 'white' as racial concepts in ambiguous contexts.

  2. Can AI Examine Novelty of Patents?: Novelty Evaluation Based on the Correspondence between Patent Claim and Prior Art

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A new patent novelty benchmark from real examiner rejections shows large language models can classify novelty at about 62% accuracy, while smaller classification models perform at chance.

  3. How well can LLMs Grade Essays in Arabic?

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Generative LLMs, including Arabic-specific ones, underperform a fine-tuned BERT model on Arabic essay scoring, with bilingual prompting giving the best LLM results.

  4. A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Large language models with in-context learning outperform fine-tuned BERT and RoBERTa for music entity detection in user-generated content, but their edge shrinks for entities not memorized during pre-training.

  5. Unified Parameter-Efficient Unlearning for LLMs

    cs.AI 2024-11 conditional novelty 6.0 of 10

    An influence-function-based parameter editing framework performs instance removal, query modification, and response correction on PEFT adapters without retraining.

  6. RetrieveAll: A Multilingual Named Entity Recognition Framework with Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    RetrieveAll combines per-language LoRA adapters with retrieval of entity and context examples to improve multilingual NER, claiming an average 12.1% F1 gain on PAN-X.

  7. On LLM-Enhanced Mixed-Type Data Imputation with High-Order Message Passing

    cs.LG 2025-01 reject novelty 5.0 of 10

    UnIMP combines a large language model with a hypergraph message-passing network to impute missing numerical, categorical, and text values, reporting large gains over prior methods.

  8. Political-LLM: Large Language Models in Political Science

    cs.CL 2024-12 conditional novelty 5.0 of 10

    A survey and taxonomy of LLM applications in political science, with a case study suggesting that larger LLMs reproduce ANES 2016 voting patterns more accurately than smaller ones.

  9. A Large Language Model Approach to Identify Flakiness in C++ Projects

    cs.SE 2024-12 reject novelty 4.0 of 10

    Fine-tuned Mistral-7b, Llama2-7b, and CodeLlama-7b classify flaky test root causes in C++ and Java, with Mistral-7b reporting perfect scores on a small augmented C++ dataset.

Pith tools