First Readme

README.md

@@ -0,0 +1,180 @@
+---
+language:
+- en
+library_name: vllm
+pipeline_tag: text-generation
+tags:
+  - text-generation
+  - conversational
+  - compressed-tensors
+  - awq
+  - w4a16
+  - quantized
+  - moe
+base_model: PrimeIntellect/INTELLECT-3
+base_model_relation: quantized
+quantized_by: TheHouseOfTheDude
+license: other
+---
+
+# INTELLECT-3 — **Quantized** (compressed-tensors for vLLM, GLM-4.5-Air MoE finetune)
+
+This repository provides **quantized runtime builds** of  
+**PrimeIntellect/INTELLECT-3**, repackaged for **vLLM** using the **compressed-tensors** format.
+
+> **TL;DR**
+> - **Quantized** branch: **W4A16** (INT4 weights / A16 activations) for vLLM via `--quantization compressed-tensors`.
+> - Same calibration recipe as our recent cards: **512** chat samples at **2048** tokens max from **`neuralmagic/LLM_compression_calibration`** (rendered with the model’s chat template).
+> - Weight-only **AWQ**, **group size 128**, **symmetric** quant, `lm_head` left in higher precision, exported with `save_compressed=True`.
+> - Parent is a **GLM-4.5-Air MoE** finetune; notes below cover MoE-specific considerations.
+
+---
+
+## Revisions & Branches
+
+> The **`main`** branch is a landing page (model card + links). Runnable artifacts live in per-quant branches.
+
+- **main** — placeholder / landing page  
+- **W4A16** — 4-bit weights / 16-bit activations (compressed-tensors)
+
+**Quick links**
+
+- main: https://huggingface.co/TheHouseOfTheDude/INTELLECT-3_Compressed-Tensors/tree/main  
+- W4A16: https://huggingface.co/TheHouseOfTheDude/INTELLECT-3_Compressed-Tensors/tree/W4A16
+
+---
+
+## What’s inside (per revision)
+
+- Sharded **quantized** weights (`*.safetensors`) + index (`model.safetensors.index.json`)  
+- `config.json` with **compressed-tensors** metadata (`weight_format`, `quantization`, `quantization_config`, etc.)  
+- Tokenizer artifacts (`tokenizer.json`, `tokenizer.model`, merges/vocab as applicable)  
+- Optional: `chat_template.jinja` (inherits the finetune’s chat style)
+
+> Exact file lists may differ between branches — see **Files and versions** for each revision.
+
+---
+
+## Quantization & calibration details (same script/recipe family as previous card)
+
+**Method / flow**
+- `llmcompressor` **oneshot** pipeline with an **AWQModifier** (weight-only).
+
+**Targets / exclusions**
+- Quantize **Linear** layers across the model (including MoE expert linear projections).  
+- **Ignore** `lm_head` (kept in higher precision).
+
+**Weights / grouping**
+- **INT4** (`num_bits=4`, `type="int"`, `symmetric=True`)  
+- Strategy: `"group"` with **`group_size=128`** (Marlin-friendly)  
+- **Activations are not quantized** (runtime **A16**: BF16/FP16)
+
+**Calibration dataset & preprocessing**
+- Dataset: **`neuralmagic/LLM_compression_calibration`**, split **`train`**  
+- **NUM_CALIBRATION_SAMPLES = 512** (random subset with fixed seed)  
+- **MAX_SEQUENCE_LENGTH = 2048**  
+- Each sample’s `messages` list is rendered via the model tokenizer’s  
+  `apply_chat_template(..., tokenize=False)`, then tokenized with:
+  - `max_length=2048`, `truncation=True`, `padding=False`, `add_special_tokens=False`
+
+**Compression call**
+- `oneshot(..., max_seq_length=2048, num_calibration_samples=512, tokenizer=tokenizer)` on the preprocessed dataset
+
+**Export for vLLM**
+- Saved with **`save_compressed=True`** so **vLLM** reads the **compressed-tensors** runtime layout directly
+
+---
+
+## GLM-4.5-Air MoE notes
+
+- **Mixture-of-Experts (MoE)** means most transformer blocks host multiple expert FFNs with a router/gating network that activates a subset per token.
+- **Quantization impact:** AWQ weight-only quantization is applied to expert **Linear** layers as well as shared projections; the **router** (small linear(s)) is quantized like other Linear layers.
+- **Serving tips (vLLM):**
+  - Ensure your vLLM build supports MoE routing for the GLM-family architecture.
+  - Throughput depends on **expert parallelism** + **tensor parallelism**; scale `--tensor-parallel-size` to your GPUs and mind interconnect bandwidth.
+  - Token-wise active experts increase **KV-cache** and memory pressure slightly; keep `--max-model-len` aligned with hardware.
+
+---
+
+## Context length
+
+- **Calibration context:** up to **2048 tokens** per sample (as above).  
+- **Model context window:** inherited from **PrimeIntellect/INTELLECT-3**; quantization does **not** change rope/position encodings—only the numeric representation of the weights.
+
+---
+
+## Quickstart — vLLM (compressed-tensors)
+
+Install vLLM (recent version recommended):
+
+    pip install vllm
+
+Serve (adjust to your hardware):
+
+    CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
+    vllm serve TheHouseOfTheDude/INTELLECT-3_Compressed-Tensors \
+      --quantization compressed-tensors \
+      --tensor-parallel-size 8 \
+      --max-model-len 2048 \
+      --gpu-memory-utilization 0.70 \
+      --dtype bfloat16
+
+Example Chat Completions:
+
+    curl http://localhost:8000/v1/chat/completions \
+      -H "Content-Type: application/json" \
+      -d '{
+        "model": "TheHouseOfTheDude/INTELLECT-3_Compressed-Tensors",
+        "messages": [
+          {"role":"system","content":"You are INTELLECT — helpful, precise, and safe."},
+          {"role":"user","content":"Outline a plan for multi-document retrieval with MoE models."}
+        ],
+        "max_tokens": 512,
+        "temperature": 0.7,
+        "top_p": 0.95
+      }'
+
+> **Note:** `compressed-tensors` is a **vLLM runtime** format. Loading directly with vanilla 🤗 Transformers is **not supported**.  
+> For Transformers, use a compatible export (e.g., GPTQ/AWQ for Transformers) or the full-precision finetune.
+
+---
+
+## Prompting / chat template
+
+This package follows the **finetuned parent’s** chat conventions. If a `chat_template.jinja` is present, libraries that support `apply_chat_template` will automatically format messages.
+
+Guidelines:
+- Keep the **system** message concise (behavior, tone, safety constraints).  
+- Provide clear **user** instructions; for multi-step tasks, list steps explicitly.
+
+---
+
+## Intended use & safety
+
+This quantization:
+- **Does not** change underlying behavior or content tendencies.  
+- **Only** changes weight storage for efficient inference.
+
+Apply appropriate **content filters / policies** for your deployment context.
+
+---
+
+## Lineage
+
+- **Finetuned parent:** https://huggingface.co/PrimeIntellect/INTELLECT-3  
+- **This repo:** **Quantized child** of the finetune (**compressed-tensors** for vLLM)
+
+---
+
+## Hardware tips
+
+- 100B+-class MoE models benefit from **multi-GPU** tensor parallel; interconnect bandwidth matters (NVLink/IB).  
+- Long contexts are **KV-cache** heavy — tune `--max-model-len` and batch size.  
+- Prefer **BF16** on GPUs with native support; otherwise **FP16**.  
+- Consider CUDA Graphs if stable in your environment.
+
+---
+
+## Changelog
+
+- **v1 (current)** — Initial **compressed-tensors W4A16** quantization with **512-sample / 2048-token** AWQ calibration; vLLM-ready packaging.