thsluck/llm-course-hw3-lora

Text Generation • 0.3B • Updated Apr 11

Note LoRA (Low-Rank Adaptation): LoRA reduces computational demands by updating low-rank decompositions of weight matrices during fine-tuning instead of the full matrices. This minimizes trainable parameters and preserves pre-trained weights, enabling efficient task-specific adaptation. It's particularly useful in memory and resource-constrained environments.

thsluck/llm-course-hw3-dora

Text Generation • 0.3B • Updated Apr 11

Note DoRA (Weight-Decomposed Low-Rank Adaptation): DoRA fine-tunes large models by decomposing weight updates into low-rank components, separating magnitude and direction. This targeted adaptation reduces computational and memory demands, making it ideal for resource-constrained environments while maintaining performance.

thsluck/llm-course-hw3-tinyllama-qlora

Updated Apr 11

Note QLoRA (Quantized Low-Rank Adaptation): QLoRA efficiently fine-tunes large models by integrating weight quantization with low-rank decomposition. Quantization lowers memory usage by using reduced precision, while low-rank adaptation targets essential weight components. This combination significantly cuts down on memory and computational needs, maintaining accuracy and making it ideal for environments with limited resources.