Llama-2 LoRA Fine-tuning
What it is
A repository providing a pipeline for fine-tuning Llama-2-Chat models using Low-Rank Adaptation (LoRA) and DeepSpeed. It includes scripts for model downloading, Chinese vocabulary expansion, and training with support for 4-bit and 8-bit quantization.
The problem it solves
High memory requirements for fine-tuning Llama-2 and the complexity of handling vocabulary expansion for multilingual support.
What you could use it for
- Fine-tuning Llama-2 models on custom datasets using LoRA
- Training LLMs with limited VRAM (16GB+) using DeepSpeed and bitsandbytes
- Expanding Llama-2's vocabulary to support Chinese characters
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.