---
base_model:
- mistralai/Mixtral-8x22B-v0.1
base_model_relation: quantized
pipeline_tag: text-generation
tags:
  - quantized
  - hardware-optimized
  - mixtral
  - tensordyne
license: apache-2.0
---

## 📝 Overview
Tensordyne builds advanced [AI-inference systems](https://www.tensordyne.ai/inference-system), enabling faster, more affordable, and sustainable generative AI.  

This repository provides resources to quickly get started with **[Mixtral-8x22B](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1)** on the **Tensordyne Inference System and its SDK**.  

## 🧩 Model Details
- **Quantization:** post-training quantization of the base model, no fine-tuning or additional training was performed
- **Supported data types:** Tensordyne FP16 (tFP16), Tensordyne FP8 (tFP8), mixed-precision

## ⚙️ Quantization
The Tensordyne SDK offers multiple post-training quantization strategies to convert AI models for efficient inference on the Tensordyne Inference System — fully customizable for your optimization targets.  
We showcase several preselected quantization variants that can be applied on-the-fly to quantize to Tensordyne data types here. The calibration-based strategies are defined by quantization configurations provided as `.json`.

The quantized models are evaluated on 10% of the [WikiText-2 raw v1](https://huggingface.co/datasets/Salesforce/wikitext) test set. Negative relative perplexity drops indicate that the model performs better than the float base model.

| Model Configuration              | Absolute Perplexity | Relative Perplexity Drop vs. BF16 | Details                                                     |
|----------------------------------|---------------------|-----------------------------------|-------------------------------------------------------------|
| BF16                             | 2.923               | –                                 | The baseline model trained in BF16                          |
| calibration_free_tFP16           | 2.921               | -0.05 %                          | calibration-free tFP16 quantization                         |
| calibration_based_tFP16          | 2.923               | 0.00 %                           | calibration-based tFP16 quantization                        |
| layerwise_mixed_precision        | 2.932               | 0.30 %                           | calibration-based mixed-precision: tFP8, outliers in tFP16  |
| calibration_free_dynamic_tFP8    | 2.926               | 0.13 %                           | calibration-free tFP8 dynamic quantization                 |

## 🚀 Getting Started
Refer to the [Tensordyne Hugging Face Hub tutorial](https://resources.tensordyne.ai/sdk/v0.1.1/tutorials/tutorials/#tensordyne-hugging-face-hub-tutorials) for instructions on using the artifacts provided in this repository.  
Our [hosted documentation](https://resources.tensordyne.ai/sdk/v0.1.1/) provides more information on Tensordyne's quantization strategies and introduces you to our SDK.