Create README.md

README.md

@@ -0,0 +1,72 @@
+---
+datasets:
+- HuggingFaceFW/fineweb-edu
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+---
+
+# DMaS-LLaMa-Lite-step-100
+
+This repository provides access to **DMaS-LLaMa-Lite-step-100**, a 1.7-billion-parameter language model based on the LLaMa architecture. The model has been trained from scratch as part of the DMaS-LLaMa-Lite project using approximately 20 billion tokens of high-quality educational content. 
+
+## Model Overview
+
+- **Architecture**: LLaMa-based
+- **Parameters**: 1.7B (36 layers, 32 attention heads, RMSNorm)
+- **Tokenizer**: GPT-2 tokenizer
+- **Training Data**: FineWeb-Edu subset (educational text)
+- **Training Steps**: 100
+- **Optimizer**: AdamW with linear warmup and decay
+- **Hardware**: Trained on 1-2 RTX A6000 GPUs with PyTorch DDP
+- **Dataset Source**: [FineWeb-Edu Dataset](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
+
+The training process emphasizes qualitative improvements in coherence, fluency, and factual grounding, demonstrating competitive results even with fewer tokens compared to larger-scale models.
+
+
+This checkpoint represents the model's state at **100 training steps**. Validation loss and downstream performance benchmarks demonstrate notable early improvements in text fluency and alignment with prompts.
+
+
+## Training Code
+
+The training script, including configurations and instructions, is open-sourced and available here:  
+📄 **[DMaS-LLaMa-Lite Training Code](https://github.com/McGill-DMaS/DMaS-LLaMa-Lite-Training-Code)**
+
+## Usage
+
+You can load the model with Hugging Face Transformers library:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "McGill-DMaS/DMaS-LLaMa-Lite-step-100"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+
+inputs = tokenizer("The Pyramids of Giza in Egypt are some of the oldest man-made structures in the world.", return_tensors="pt")
+outputs = model.generate(**inputs, max_length=50)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Citation
+
+If you use this model or its training insights in your work, please cite the following [paper](https://arxiv.org/abs/2412.13335):
+
+```bibtex
+@inproceedings{LFH25ijcnn,
+        author = "M. Q. Li and B. C. M. Fung and S.-C. Huang",
+        title = "Training dynamics of a 1.7B {LLaMa} model: a data-efficient approach",
+        booktitle = "Proc. of the IEEE International Joint Conference on Neural Networks (IJCNN),
+        pages = "",
+        address = "Rome, Italy",
+        month = "July",
+        year = "2025",
+        publisher = "IEEE Press",
+}
+```
+
+## License
+
+This model and code are released under the **Apache License 2.0**. Please check the respective repositories for detailed terms.