neuralsrg/llm-course-hw3-dora

Text Generation • 0.3B • Updated Apr 11

Note DoRA (Dynamic Reparameterization Adjustment): DoRA takes a step beyond static parameter tuning by dynamically adjusting model parameters during training. It focuses on finding an optimal parameter configuration on-the-fly, which can respond to the complexities of the data and task requirements encountered during the training process. DoRA's dynamic approach offers advantages in handling diverse datasets and varying task complexities without the need for manual tuning.

neuralsrg/llm-course-hw3-lora

Text Generation • 0.3B • Updated Apr 11

Note LoRA (Low-Rank Adaptation): LoRA is a parameter-efficient fine-tuning technique designed to improve the scalability of large language models. Instead of updating all the parameters of a pre-trained model, LoRA focuses on injecting low-rank updates into the model's weight matrices. By limiting the extent of parameter changes, LoRA reduces the computational cost and memory footprint associated with training, while maintaining the model's performance.

neuralsrg/llm-course-hw3-tinyllama-qlora

Updated Apr 11

Note QLoRA (Quantized Low-Rank Adaptation): QLoRA extends the concept of low-rank adaptation by incorporating model quantization. In this approach, model weights are not only adapted using low-rank updates but are also quantized to reduce their precision. This dual strategy significantly reduces the memory and storage requirements of large models. QLoRA enables the deployment of sophisticated models on resource-constrained hardware without sacrificing performance.