{"work":{"id":"462b3287-e058-48e3-b5e3-82a5f2a8dc06","openalex_id":"https://openalex.org/W4393108840","doi":"10.48550/arxiv.2403.13372","arxiv_id":"2403.13372","raw_key":null,"title":"LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models","authors":null,"authors_text":"Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng","year":2024,"venue":"cs.CL","abstract":"Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, it requires non-trivial efforts to implement these methods on different models. We present LlamaFactory, a unified framework that integrates a suite of cutting-edge efficient training methods. It provides a solution for flexibly customizing the fine-tuning of 100+ LLMs without the need for coding through the built-in web UI LlamaBoard. We empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. It has been released at https://github.com/hiyouga/LLaMA-Factory and received over 25,000 stars and 3,000 forks.","external_url":"https://arxiv.org/abs/2403.13372","cited_by_count":13,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2403.13372","created_at":"2026-05-09T06:15:38.801427+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models","render_title":"LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models"},"hub":{"state":{"work_id":"462b3287-e058-48e3-b5e3-82a5f2a8dc06","tier":"super_hub","tier_reason":"100+ Pith inbound or 10,000+ external citations","pith_inbound_count":105,"external_cited_by_count":13,"distinct_field_count":13,"first_pith_cited_at":"2024-08-28T05:53:46+00:00","last_pith_cited_at":"2026-07-08T23:54:36+00:00","author_build_status":"needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-18T23:19:26.183439+00:00","tier_text":"super_hub"},"tier":"super_hub","role_counts":[{"context_role":"background","n":7},{"context_role":"method","n":6}],"polarity_counts":[{"context_polarity":"background","n":7},{"context_polarity":"use_method","n":6}],"runs":{"ask_index":{"job_type":"ask_index","status":"succeeded","result":{"title":"LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models","claims":[{"claim_text":"Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, it requires non-trivial efforts to implement these methods on different models. We present LlamaFactory, a unified framework that integrates a suite of cutting-edge efficient training methods. It provides a solution for flexibly customizing the fine-tuning of 100+ LLMs without the need for coding through the built-in web UI LlamaBoard. We empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. It has been released at https://githu","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"of the policy and reference models, respectively. The reward ri =R(o i) and the advantages ˆAi,t = ri−mean(r1,...,rG) std(r1,...,rG) are calculated using the normalized rewards within each group following GRPO.D KL denotes the estimated KL-divergence [54]. 4 Implementation During the Python-IR pivoted supervised fine-tuning (SFT) stage, we use LLaMAFactory [90] to perform SFT of the open-source base model Qwen2.5-Coder-7B-Instruct on DPY2OTHERS-SFT . We train the model for 3 epochs with a learni","claim_type":"method","confidence":0.95,"evidence_strength":"citation_context"},{"claim_text":"vironments, primarily Claude Code [44], consistent with the AI-native devel- opment methodology described in our earlier work [26]. This approach signif- icantly reduced implementation overhead and enabled rapid experimentation and short feedback loops throughout the development process. The fine-tuning of domain-specialized LLMs was conducted using the Unsloth library [45, 46] on NVIDIA A100 GPUs via Google Colab [47]. To ensure efficient training within resource-constrained environments, Low- ","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"similarity (J ), contour accuracy (F), and their composite measure (J&F ). The J metric computes the intersection-over-union (IoU) between the predicted mask sequence and the ground truth, while theFmetric assesses boundary alignment precision based on contour matching. 4.2 Implementation details During the supervised fine-tuning (SFT) stage, we utilize the LLaMA-Factory framework [33] to fine- tune the Qwen2.5-VL-3B model with LoRA [10](rank=8) while freezing all other parameters. The training ","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"then conduct training analysis and ablation studies in Sec. 5.3 and Sec. 5.4. Finally, we assess the cross-language generalization of DSC to Python (Sec. 5.5) and present human/case analyses (Sec. 5.6). Additional details are provided in Appendix. 5.1 Experimental Setup Training details.We use LLaMA-Factory [58] for supervised fine-tuning and EasyR1 [59], built upon verl [39], for reinforcement learning. For SFT, the learning rate is set to 5 × 10−5, with a total batch size of 128, a maximum tok","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URLhttp://arxiv.org/abs/2403.13372. [94] Jialong Zhou, Lichao Wang, and Xiao Yang. GUARDIAN: Safeguarding LLM multi-agent col- laborations with temporal graph modeling.arXiv preprint arXiv:2505.19234, 2025. URL https://arxiv.org/abs/2505.19234. [95] Yujun Zhou, Yue Huang, Han Bao, Kehan Guo, Zhenwen Liang, P","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Each sample in- cludes an image, a question, a ground-truth answer, and a bounding box marking the image regions relevant to the question. Further dataset statistics and curation details are provided in the Appendix. Implementation Details.Our main experiments utilize Qwen2.5-VL [6] (3B and 7B) as the base model. SFT is trained using llama-factory [96] and GRPO is trained with the TRL framework [72]. We use a total batch size of 64 with mixed precision. The KL divergence coefficient is 0.001, an","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (7 contexts).","role_counts":[{"n":7,"context_role":"background"},{"n":6,"context_role":"method"}]},"error":null,"updated_at":"2026-07-02T21:43:01.865306+00:00"},"author_expand":{"job_type":"author_expand","status":"succeeded","result":{"authors_linked":[{"id":"abad0a97-eca5-4e1c-90e7-ed22d8e3dcfb","orcid":null,"display_name":"Yaowei Zheng"},{"id":"7be9a633-1067-4c3f-b452-412339b25cf9","orcid":null,"display_name":"Richong Zhang"},{"id":"fc906d85-9ba1-4c86-ba1f-a43f83c19b5c","orcid":null,"display_name":"Junhao Zhang"},{"id":"e2b3f260-4a4b-45bf-808d-f01a96197606","orcid":null,"display_name":"Yanhan Ye"},{"id":"7b48698c-c0c7-48c0-b8c8-abb6163668a3","orcid":null,"display_name":"Zheyan Luo"},{"id":"97ba0ea6-4fc8-4530-8dfa-ee88d41b969f","orcid":null,"display_name":"Zhangchi Feng"}]},"error":null,"updated_at":"2026-07-02T21:43:01.859384+00:00"},"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-14T17:26:24.732551+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":17},{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":14},{"title":"Qwen2.5-VL Technical Report","work_id":"69dffacb-bfe8-442d-be86-48624c60426f","shared_citers":13},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":9},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":8},{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":8},{"title":"Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling","work_id":"ee70bdc8-4656-4849-ada7-ce42a2278d70","shared_citers":7},{"title":"Qwen3-VL Technical Report","work_id":"1fe243aa-e3c0-4da6-b391-4cbcfc88d5c0","shared_citers":7},{"title":"HybridFlow: A Flexible and Efficient RLHF Framework","work_id":"7eb9c9f4-b322-4bba-8011-09ff8d6ad801","shared_citers":6},{"title":"LoRA: Low-Rank Adaptation of Large Language Models","work_id":"0426219a-789e-4964-adc8-a04538510818","shared_citers":6},{"title":"Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution","work_id":"8abcfe4f-e0fb-44b7-9123-448fac95f90a","shared_citers":6},{"title":"InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency","work_id":"b8f5e260-fff5-444e-bcf5-2c42cfefd83d","shared_citers":5},{"title":"LLaVA-OneVision: Easy Visual Task Transfer","work_id":"f5f2452b-f2a9-49ac-b38d-c76e18cdfe49","shared_citers":5},{"title":"Qwen2.5 Technical Report","work_id":"d8432992-4980-4a81-85c7-9fa2c2b87f85","shared_citers":5},{"title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","work_id":"64019d00-0b11-4bbd-b173-b46c8fad0157","shared_citers":4},{"title":"GPT-4o System Card","work_id":"f37bf1c7-4964-4e56-9762-d20da8d9009f","shared_citers":4},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":4},{"title":"InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models","work_id":"fe8637aa-12bc-4434-8d36-9f57b5eebcbe","shared_citers":4},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":4},{"title":"A Survey of Large Language Models","work_id":"de1b42b5-4a0a-4b1f-8c78-1f7fe21be6c9","shared_citers":3},{"title":"Decoupled Weight Decay Regularization","work_id":"07ef7360-d385-4033-83f7-8384a6325204","shared_citers":3},{"title":"Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities","work_id":"008df105-2fdd-45d8-857a-8e35868aecb6","shared_citers":3},{"title":"Gemma 3 Technical Report","work_id":"f93e08bf-9e96-409b-8ac6-b8385fd17fd7","shared_citers":3},{"title":"gpt-oss-120b & gpt-oss-20b Model Card","work_id":"178c1f7e-4f19-4392-a45d-45a6dfa88ead","shared_citers":3}],"time_series":[{"n":42,"year":2026}],"dependency_candidates":[]},"error":null,"updated_at":"2026-05-14T17:26:46.397682+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-14T17:26:37.718192+00:00"},"role_polarity":{"job_type":"role_polarity","status":"succeeded","result":{"title":"LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models","claims":[{"claim_text":"Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, it requires non-trivial efforts to implement these methods on different models. We present LlamaFactory, a unified framework that integrates a suite of cutting-edge efficient training methods. It provides a solution for flexibly customizing the fine-tuning of 100+ LLMs without the need for coding through the built-in web UI LlamaBoard. We empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. It has been released at https://githu","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"of the policy and reference models, respectively. The reward ri =R(o i) and the advantages ˆAi,t = ri−mean(r1,...,rG) std(r1,...,rG) are calculated using the normalized rewards within each group following GRPO.D KL denotes the estimated KL-divergence [54]. 4 Implementation During the Python-IR pivoted supervised fine-tuning (SFT) stage, we use LLaMAFactory [90] to perform SFT of the open-source base model Qwen2.5-Coder-7B-Instruct on DPY2OTHERS-SFT . We train the model for 3 epochs with a learni","claim_type":"method","confidence":0.95,"evidence_strength":"citation_context"},{"claim_text":"vironments, primarily Claude Code [44], consistent with the AI-native devel- opment methodology described in our earlier work [26]. This approach signif- icantly reduced implementation overhead and enabled rapid experimentation and short feedback loops throughout the development process. The fine-tuning of domain-specialized LLMs was conducted using the Unsloth library [45, 46] on NVIDIA A100 GPUs via Google Colab [47]. To ensure efficient training within resource-constrained environments, Low- ","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"similarity (J ), contour accuracy (F), and their composite measure (J&F ). The J metric computes the intersection-over-union (IoU) between the predicted mask sequence and the ground truth, while theFmetric assesses boundary alignment precision based on contour matching. 4.2 Implementation details During the supervised fine-tuning (SFT) stage, we utilize the LLaMA-Factory framework [33] to fine- tune the Qwen2.5-VL-3B model with LoRA [10](rank=8) while freezing all other parameters. The training ","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"then conduct training analysis and ablation studies in Sec. 5.3 and Sec. 5.4. Finally, we assess the cross-language generalization of DSC to Python (Sec. 5.5) and present human/case analyses (Sec. 5.6). Additional details are provided in Appendix. 5.1 Experimental Setup Training details.We use LLaMA-Factory [58] for supervised fine-tuning and EasyR1 [59], built upon verl [39], for reinforcement learning. For SFT, the learning rate is set to 5 × 10−5, with a total batch size of 128, a maximum tok","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URLhttp://arxiv.org/abs/2403.13372. [94] Jialong Zhou, Lichao Wang, and Xiao Yang. GUARDIAN: Safeguarding LLM multi-agent col- laborations with temporal graph modeling.arXiv preprint arXiv:2505.19234, 2025. URL https://arxiv.org/abs/2505.19234. [95] Yujun Zhou, Yue Huang, Han Bao, Kehan Guo, Zhenwen Liang, P","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"Each sample in- cludes an image, a question, a ground-truth answer, and a bounding box marking the image regions relevant to the question. Further dataset statistics and curation details are provided in the Appendix. Implementation Details.Our main experiments utilize Qwen2.5-VL [6] (3B and 7B) as the base model. SFT is trained using llama-factory [96] and GRPO is trained with the TRL framework [72]. We use a total batch size of 64 with mixed precision. The KL divergence coefficient is 0.001, an","claim_type":"method","confidence":0.9,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (7 contexts).","role_counts":[{"n":7,"context_role":"background"},{"n":6,"context_role":"method"}]},"error":null,"updated_at":"2026-07-02T21:43:01.862813+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models","claims":[{"claim_text":"Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, it requires non-trivial efforts to implement these methods on different models. We present LlamaFactory, a unified framework that integrates a suite of cutting-edge efficient training methods. It provides a solution for flexibly customizing the fine-tuning of 100+ LLMs without the need for coding through the built-in web UI LlamaBoard. We empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. It has been released at https://githu","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-14T17:26:28.871266+00:00"}},"summary":{"title":"LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models","claims":[{"claim_text":"Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, it requires non-trivial efforts to implement these methods on different models. We present LlamaFactory, a unified framework that integrates a suite of cutting-edge efficient training methods. It provides a solution for flexibly customizing the fine-tuning of 100+ LLMs without the need for coding through the built-in web UI LlamaBoard. We empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. It has been released at https://githu","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":17},{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":14},{"title":"Qwen2.5-VL Technical Report","work_id":"69dffacb-bfe8-442d-be86-48624c60426f","shared_citers":13},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":9},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":8},{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":8},{"title":"Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling","work_id":"ee70bdc8-4656-4849-ada7-ce42a2278d70","shared_citers":7},{"title":"Qwen3-VL Technical Report","work_id":"1fe243aa-e3c0-4da6-b391-4cbcfc88d5c0","shared_citers":7},{"title":"HybridFlow: A Flexible and Efficient RLHF Framework","work_id":"7eb9c9f4-b322-4bba-8011-09ff8d6ad801","shared_citers":6},{"title":"LoRA: Low-Rank Adaptation of Large Language Models","work_id":"0426219a-789e-4964-adc8-a04538510818","shared_citers":6},{"title":"Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution","work_id":"8abcfe4f-e0fb-44b7-9123-448fac95f90a","shared_citers":6},{"title":"InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency","work_id":"b8f5e260-fff5-444e-bcf5-2c42cfefd83d","shared_citers":5},{"title":"LLaVA-OneVision: Easy Visual Task Transfer","work_id":"f5f2452b-f2a9-49ac-b38d-c76e18cdfe49","shared_citers":5},{"title":"Qwen2.5 Technical Report","work_id":"d8432992-4980-4a81-85c7-9fa2c2b87f85","shared_citers":5},{"title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","work_id":"64019d00-0b11-4bbd-b173-b46c8fad0157","shared_citers":4},{"title":"GPT-4o System Card","work_id":"f37bf1c7-4964-4e56-9762-d20da8d9009f","shared_citers":4},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":4},{"title":"InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models","work_id":"fe8637aa-12bc-4434-8d36-9f57b5eebcbe","shared_citers":4},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":4},{"title":"A Survey of Large Language Models","work_id":"de1b42b5-4a0a-4b1f-8c78-1f7fe21be6c9","shared_citers":3},{"title":"Decoupled Weight Decay Regularization","work_id":"07ef7360-d385-4033-83f7-8384a6325204","shared_citers":3},{"title":"Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities","work_id":"008df105-2fdd-45d8-857a-8e35868aecb6","shared_citers":3},{"title":"Gemma 3 Technical Report","work_id":"f93e08bf-9e96-409b-8ac6-b8385fd17fd7","shared_citers":3},{"title":"gpt-oss-120b & gpt-oss-20b Model Card","work_id":"178c1f7e-4f19-4392-a45d-45a6dfa88ead","shared_citers":3}],"time_series":[{"n":42,"year":2026}],"dependency_candidates":[]},"authors":[{"id":"fc906d85-9ba1-4c86-ba1f-a43f83c19b5c","orcid":null,"display_name":"Junhao Zhang","source":"manual","import_confidence":0.72},{"id":"7be9a633-1067-4c3f-b452-412339b25cf9","orcid":null,"display_name":"Richong Zhang","source":"manual","import_confidence":0.72},{"id":"e2b3f260-4a4b-45bf-808d-f01a96197606","orcid":null,"display_name":"Yanhan Ye","source":"manual","import_confidence":0.72},{"id":"abad0a97-eca5-4e1c-90e7-ed22d8e3dcfb","orcid":null,"display_name":"Yaowei Zheng","source":"manual","import_confidence":0.72},{"id":"97ba0ea6-4fc8-4530-8dfa-ee88d41b969f","orcid":null,"display_name":"Zhangchi Feng","source":"manual","import_confidence":0.72},{"id":"7b48698c-c0c7-48c0-b8c8-abb6163668a3","orcid":null,"display_name":"Zheyan Luo","source":"manual","import_confidence":0.72}]}}