---
base_model: orai-nlp/Llama3.2-3B-eu-continual
language:
- eu
library_name: transformers
pipeline_tag: text-generation
license: llama3.2
---

# Llama-eus-3B-DIPLomA

**Basque Instruction-Following Model using DIPLomA Delta Merging**

`llama-eus-3B-DIPLomA` is a large language model derived from [Llama3.2-eu-3B-continual](https://huggingface.co/orai-nlp/Llama3.2-3B-eu-continual) that introduces **instruction-following capabilities** in Basque through the **DIPLomA (Decoupled Instruction-Preserving Language Adaptation)** method. It combines the **Basque language knowledge** of *Llama3.2-eu-3B-continual*, continually pretrained on Basque, with the **instruction-tuned behavior** of *Meta-Llama-3.2-1B-Instruct* using a lightweight **delta-merging technique** that transfers instruction alignment without costly retraining.

📕 Paper: [DIPLomA: Efficient Adaptation of Instructed LLMs to Low-Resource Languages via Post-Training Delta Merging](https://aclanthology.org/2025.findings-emnlp.1355.pdf)

## Model details

| Attribute | Description |
|------------|--------------|
| **Model name** | `llama-eus-3B-DIPLomA` |
| **Base model** | [Llama3.2-eu-3B-continual](https://huggingface.co/orai-nlp/Llama3.2-3B-eu-continual) |
| **Instruction delta from** | [Meta-Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) and [Meta-Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B) |
| **Architecture** | LLaMA 3.2 (3B parameters) |
| **Tokenizer** | LLaMA 3.2 tokenizer |
| **Languages** | Basque (eu), English (en) |
| **Training method** | DIPLomA (post-training delta merging) |
| **License** | Llama 3.2 Community License |
| **Developed by** | Orai NLP Technologies |

## Overview

**Llama-eus-3B-DIPLomA** brings *instruction-following capabilities* to the Basque language by transferring alignment knowledge from an English instruction-tuned model.

Instead of retraining the full model with instruction data in Basque, DIPLomA:
1. Starts from the **language-adapted model**,
2. Computes the **parameter delta** between the instruct adapted model and its base counterpart,
3. **Merges** that delta into the Basque model.

This preserves the **Basque fluency** while importing the **instructional alignment and safety** of the English model.

## DIPLomA Method

**DIPLomA (Decoupled Instruction-Preserving Language Adaptation)** is a two-stage approach:

1. **Continual pretraining**: adapt the base model to Basque using a mixed Basque-English corpus to maintain multilingual ability.
2. **Instruction delta merging**: apply the difference `(Winstruct − Wbase)` from the instructed model onto the Basque-adapted weights: W_DIPLOmA = W_LangAdapted + α · (Winstruct − Wbase)

Where **α** (alpha) controls the trade-off between language retention and instruction-following strength.

## Performance summary

  
| Models                       | **EU** Instr. Follow. | **EU** Safety | **EN** Inst. Follow. | **EN** Safety |
|------------------------------|-----------------------|---------------|----------------------|---------------|
|  **Llama-eus-3B-DIPLomA**    |        **40**         |    **44**     |          70          |      93       |
| Llama-3.2-3B-Instruct        |           4           |      4        |        **79**        |    **98**     |

*(see the DIPLomA [paper](https://aclanthology.org/2025.findings-emnlp.1355.pdf) for furher detailed evaluation results)*

The model shows improvements in **instruction understanding and task completion** in Basque while maintaining good multilingual generalization.

## Technical summary

- Architecture: LLaMA 3.2, 28 transformer layers, 3072 hidden size, 24 heads
- Sequence length: 4096 tokens
- Training hardware: 8× A100 80GB GPUs
- Continual pretraining corpus: 531 M Basque words (ZelaiHandi) + 300 M English tokens (FineWeb subset)
- Optimizer: AdamW, cosine LR schedule
- DIPLomA merge coefficient (α): 1.0

## License

This model inherits the Llama 3.2 Community License from its base model. Before use or redistribution, please review the license terms.

## Citation

If you use Llama-eus-3B-DIPLomA please cite the following reference:

```bibtex
@inproceedings{sarasua-etal-2025-diploma,
    title = "{DIPL}om{A}: Efficient Adaptation of Instructed {LLM}s to Low-Resource Languages via Post-Training Delta Merging",
    author = "Sarasua, Ixak  and
      Corral, Ander  and
      Saralegi, Xabier",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-emnlp.1355/",
    pages = "24898--24912",
    ISBN = "979-8-89176-335-7",
    abstract = "This paper investigates how open-weight instruction-tuned large language models (LLMs) can be efficiently adapted to low-resource languages without requiring costly large-scale post-training. We introduce DIPLomA (Decoupled Instruction-Preserving Language Adaptation), a lightweight delta-based transfer strategy that provides a practical and effective solution for this scenario. DIPLomA decouples language adaptation from post-training alignment by first continually pretraining a foundational LLM on a modest amount of monolingual target-language data while anchoring on English replay, and then injecting instruction-following capabilities via delta-based weight merging from the instructed counterpart of the base LLM. We evaluate DIPLomA on Basque and validate its generality on Welsh and Swahili, demonstrating consistent and substantial gains in instruction-following, linguistic proficiency, and safety. Compared to strong baselines, our method achieves average relative improvements of 50 points in Basque, 63 in Welsh, and 51 in Swahili, while preserving the original model{'}s multilingual performance. These results highlight DIPLomA as an effective, resource-efficient strategy for bringing high-quality instruction alignment to underrepresented languages at scale."
}
```

## Contact

- Ixak Sarasua (i.sarasua@orai.eus)
- Ander Corral (a.corral@orai.eus)
- Xabier Saralegi (x.saralegi@orai.eus)