metadata
language:
- en
license: mit
library_name: transformers
tags:
- text-generation
- pytorch
- custom-architecture
- rope
- rmsnorm
- swiglu
- flash-attention
- 16k-context
pipeline_tag: text-generation
widget:
- text: The future of artificial intelligence is
example_title: AI Future
- text: Write a short story about
example_title: Story Generation
- text: 'Explain quantum computing in simple terms:'
example_title: Technical Explanation
datasets:
- tiiuae/falcon-refinedweb
metrics:
- perplexity
model-index:
- name: MAP-NEO Mini
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: RefinedWeb (100K subset)
type: tiiuae/falcon-refinedweb
metrics:
- type: perplexity
value: 3.9
name: Final Training Loss
MAP-NEO Mini
Model Description
MAP-NEO Mini is a 253M parameter autoregressive language model built from scratch with modern architectural improvements. It demonstrates that high-quality language models can be trained efficiently on modest hardware while achieving competitive performance through careful data curation and architectural choices.
- Developed by: Antony Austin
- Model type: Autoregressive Language Model
- Language(s): English
- License: MIT
- Architecture: Custom transformer with RoPE, RMSNorm, SwiGLU, and Flash Attention
Key Features
- Efficient Training: Trained on RTX 5070 Laptop GPU (8GB VRAM) in ~4 hours
- Extended Context: 16,384 token context window (16x typical small models)
- Memory Efficient: Only 1.3GB VRAM for 1,800 tokens inference
- Fast Inference: ~150+ tokens/second on consumer GPU
- High Quality Data: Trained on curated RefinedWeb subset
Architecture Details
Model Architecture
- Parameters: 253,085,696 (253M)
- Layers: 16 transformer blocks
- Hidden Size: 1,024
- Attention Heads: 16
- Head Dimension: 64
- FFN Hidden Size: 2,736 (2.67x hidden size)
- Vocabulary Size: 50,257 (GPT-2 tokenizer)
- Max Sequence Length: 16,384 tokens
Architectural Innovations
- RMSNorm: Root Mean Square Layer Normalization for training stability
- RoPE: Rotary Positional Embeddings for better positional understanding
- SwiGLU: Swish-Gated Linear Units for improved FFN performance
- Flash Attention: Memory-efficient attention computation
- Weight Tying: Input/output embeddings shared for parameter efficiency
Training Data
Dataset
- Source:
tiiuae/falcon-refinedweb(curated subset) - Size: 100,000 high-quality web documents
- Tokens: ~41 million tokens
- Sequence Length: 1,024 tokens per sequence
- Sequences: 40,965 packed sequences
Data Quality
- Length filtering: 200-10,000 characters
- Language detection: English only
- Quality scoring: High-quality web content
- Deduplication: Exact and near-duplicate removal
Training Procedure
Training Configuration
- Hardware: NVIDIA RTX 5070 Laptop GPU (8GB VRAM)
- Precision: bfloat16 mixed precision
- Batch Size: 1 per device
- Gradient Accumulation: 32 steps
- Effective Batch Size: 32
- Learning Rate: 3e-4
- Scheduler: Cosine with linear warmup
- Warmup Steps: 3,750
- Total Steps: 150,000
- Training Time: ~4 hours
Optimization Details
- Optimizer: AdamW (β₁=0.9, β₂=0.95, weight_decay=0.01)
- Gradient Clipping: 1.0
- Gradient Checkpointing: Enabled for memory efficiency
- Loss Function: Cross-entropy loss
Context Extension
- Base Context: 2,048 tokens
- Extended Context: 16,384 tokens
- Method: Linear interpolation of positional embeddings
- Validation: Successfully tested up to 3,600 tokens
Performance
Training Metrics
- Final Loss: 3.907
- Training Speed: ~10 iterations/second
- Peak Memory: ~8GB VRAM
- Convergence: Smooth loss curve, no overfitting
Inference Performance
- Speed: ~150+ tokens/second (RTX 5070)
- Memory Usage: 1.3GB for 1,800 token context
- Context Limit: 3,600 tokens practical limit
- Temperature: Recommended 0.7-0.9 for creative tasks
Usage
Quick Start
import torch
from transformers import AutoTokenizer
from model_neo import NeoMini, NeoMiniConfig
# Load model
config = NeoMiniConfig()
model = NeoMini(config)
checkpoint = torch.load("extended_context_model.pt")
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
# Generate text
prompt = "The future of AI is"
input_ids = tokenizer.encode(prompt, return_tensors="pt")
with torch.no_grad():
output = model.generate(input_ids, max_length=100, temperature=0.8)
print(tokenizer.decode(output))
Interactive Chat
python interactive_chat.py
Generation Parameters
- Temperature: 0.7-0.9 for creative tasks, 0.3-0.5 for factual
- Top-k: 40-50
- Top-p: 0.8-0.9
- Repetition Penalty: 1.1-1.3
Limitations
Current Limitations
- Base Model Only: Not instruction-tuned (requires fine-tuning for chat)
- Context Window: Practical limit of ~3,600 tokens despite 16K architecture
- Hardware Requirements: Requires CUDA-capable GPU for optimal performance
- Knowledge Cutoff: Limited to web data patterns, no specific knowledge cutoff
Known Issues
- Occasionally generates repetitive patterns (fixable with fine-tuning)
- May not follow instructions well (base model behavior)
- Sometimes produces formatting artifacts from web data
Ethical Considerations
Bias and Fairness
- Trained on web data which may contain societal biases
- No explicit bias mitigation applied during training
- Users should be aware of potential biased outputs
Use Cases
Intended Uses:
- Research and experimentation
- Text generation and completion
- Creative writing assistance
- Educational purposes
Out-of-Scope Uses:
- Medical or legal advice
- High-stakes decision making
- Content that could cause harm
Environmental Impact
Carbon Footprint
- Training Hardware: Single RTX 5070 Laptop GPU (100W)
- Training Time: 4 hours
- Estimated CO₂: ~0.3 kg CO₂ equivalent
- Efficiency: 253M parameters per 0.3 kg CO₂
Model Card Authors
[Antony Austin] - Model development and training [30/08/2025] - Model card creation
Citation
@misc{mapneo_mini_2025,
title={MAP-NEO Mini: An Efficient 253M Parameter Language Model},
author={[Antony Austin]},
year={2025},
howpublished={\url{https://huggingface.co/Austin207/Map-NEO}},
note={Trained on NVIDIA RTX 5070 Laptop GPU with RefinedWeb data}
}
Technical Details
Hardware Requirements
- Minimum: 4GB VRAM for inference
- Recommended: 8GB VRAM for extended context
- Training: 8GB+ VRAM with mixed precision
- CPU: Any modern CPU (inference possible but slow)
Future Work
Planned Improvements
- Conversational fine-tuning with UltraChat dataset
- Instruction following capabilities
- Multi-language support
- Quantized versions (4-bit, 8-bit)
- ONNX export for edge deployment
Research Directions
- Context window optimization beyond 16K
- More efficient attention mechanisms
- Improved training data curation
- Specialized domain fine-tuning
Acknowledgments
- Falcon RefinedWeb: High-quality training data
- Hugging Face: Transformers library and infrastructure
- Community: Open-source ML community for architectural insights
Last Updated: August 30, 2025 Model Version: 1.0.0 Status: Base model (pre-conversational fine-tuning)