Update README.md

README.md

@@ -15,10 +15,7 @@ language:
 license: apache-2.0
 inference: false
 base_model:
-- mistralai/Ministral-3-3B-Base-2512
-extra_gated_description: >-
-  If you want to learn more about how we process your personal data, please read
-  our <a href="https://mistral.ai/terms/">Privacy Policy</a>.
+- mistralai/Ministral-3-3B-Instruct-2512
 tags:
 - mistral-common
 ---
@@ -26,11 +23,7 @@ tags:
 # Ministral 3 3B Instruct 2512
 The smallest model in the Ministral 3 family, **Ministral 3 3B** is a powerful, efficient tiny language model with vision capabilities.
 
-This model is the instruct post-trained version, fine-tuned for instruction tasks, making it ideal for chat and instruction based use cases.
-
-The Ministral 3 family is designed for edge deployment, capable of running on a wide range of hardware. Ministral 3 3B can even be deployed locally, capable of fitting in 16GB of VRAM in BF16, and less than 8GB of RAM/VRAM when quantized.
-
-We provide a no-loss FP8 version [here](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512-FP8), you can find other formats and quantizations in the [Ministral 3 - Quants](https://huggingface.co/collections/mistralai/ministral-3-quants) collection.
+The Ministral 3 family is designed for edge deployment, capable of running on a wide range of hardware. Ministral 3 3B can even be deployed locally, capable of fitting in 8GB of VRAM in FP8, and less if further quantized.
 
 ## Key Features
 Ministral 3 3B consists of two main architectural components:
@@ -63,16 +56,16 @@ Bringing advanced AI capabilities to edge and distributed environments for embed
 | Model Name                     | Type               | Precision | Link                                                                                     |
 |--------------------------------|--------------------|-----------|------------------------------------------------------------------------------------------|
 | Ministral 3 3B Base 2512       | Base pre-trained   | BF16      | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-3B-Base-2512)                |
-| **Ministral 3 3B Instruct 2512**   | **Instruct post-trained** | **BF16**   | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512)            |
+| Ministral 3 3B Instruct 2512   | Instruct post-trained | FP8   | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512)            |
 | Ministral 3 3B Reasoning 2512  | Reasoning capable  | BF16      | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-3B-Reasoning-2512)           |
 | Ministral 3 8B Base 2512       | Base pre-trained   | BF16      | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-8B-Base-2512)                |
-| Ministral 3 8B Instruct 2512   | Instruct post-trained | BF16    | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512)            |
+| Ministral 3 8B Instruct 2512   | Instruct post-trained | FP8    | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512)            |
 | Ministral 3 8B Reasoning 2512  | Reasoning capable  | BF16      | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-8B-Reasoning-2512)           |
-| Ministral 3 14B Base 2512      | Base pre-trained   | BF16      | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-14B-Base-2512)               |
-| Ministral 3 14B Instruct 2512  | Instruct post-trained | BF16    | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512)           |
+| Ministral 3 14B Base 2512      | Base pre-trained**   | BF16      | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-14B-Base-2512)               |
+| Ministral 3 14B Instruct 2512  | Instruct post-trained | FP8    | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512)           |
 | Ministral 3 14B Reasoning 2512 | Reasoning capable  | BF16      | [Hugging Face](https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512)          |
 
-Other formats available [here](https://huggingface.co/collections/mistralai/ministral-3-quants).
+Other formats available [here](https://huggingface.co/collections/mistralai/ministral-3-more).
 
 ## Benchmark Results
 
@@ -122,6 +115,407 @@ We compare Ministral 3 to similar sized models.
 | Qwen 3 4B Base      | <u>0.677</u>      | 0.405           | <u>0.570</u>   | <u>0.759</u>      | <u>0.713</u>| 0.530           |
 | Gemma 3 4B Base     | 0.516             | 0.294           | 0.430          | 0.626             | 0.589       | <u>0.640</u>    |
 
+## Usage
+
+The model can be used with the following frameworks;
+- [`vllm`](https://github.com/vllm-project/vllm): See [here](#vllm)
+- [`transformers`](https://github.com/huggingface/transformers): See [here](#transformers)
+  
+### vLLM
+
+We recommend using this model with [vLLM](https://github.com/vllm-project/vllm).
+
+#### Installation
+
+Make sure to install [`vLLM >= 0.12.0`](https://github.com/vllm-project/vllm/releases/tag/v0.12.0):
+
+```
+pip install vllm --upgrade
+```
+
+Doing so should automatically install [`mistral_common >= 1.8.6`](https://github.com/mistralai/mistral-common/releases/tag/v1.8.6).
+
+To check:
+```
+python -c "import mistral_common; print(mistral_common.__version__)"
+```
+
+You can also make use of a ready-to-go [docker image](https://github.com/vllm-project/vllm/blob/main/Dockerfile) or on the [docker hub](https://hub.docker.com/layers/vllm/vllm-openai/latest/images/sha256-de9032a92ffea7b5c007dad80b38fd44aac11eddc31c435f8e52f3b7404bbf39).
+
+#### Serve
+
+Due to their size and the FP8 format of their weights `Ministral-3-3B-Instruct-2512`, `Ministral-3-8B-Instruct-2512` and `Ministral-3-14B-Instruct-2512` can run on a single 1xH200 GPU.
+
+A simple launch command is:
+
+```bash
+vllm serve mistralai/Ministral-3-3B-Instruct-2512 \
+  --enable-auto-tool-choice --tool-call-parser mistral
+```
+
+Key parameter notes:
+
+* enable-auto-tool-choice: Required when enabling tool usage.
+* tool-call-parser mistral: Required when enabling tool usage.
+
+
+Additional flags:
+
+* You can set `--max-model-len` to preserve memory. By default it is set to `262144` which is quite large but not necessary for most scenarios.
+* You can set `--max-num-batched-tokens` to balance throughput and latency, higher means higher throughput but higher latency.
+
+#### Usage of the model
+
+Here we asumme that the model `mistralai/Ministral-3-3B-Instruct-2512` is served and you can ping it to the domain `localhost` with the port `8000` which is the default for vLLM.
+
+<details>
+  <summary>Vision Reasoning</summary>
+
+Let's see if the Ministral 3 knows when to pick a fight !
+
+```python
+from datetime import datetime, timedelta
+
+from openai import OpenAI
+from huggingface_hub import hf_hub_download
+
+# Modify OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "EMPTY"
+openai_api_base = "http://localhost:8000/v1"
+
+TEMP = 0.15
+MAX_TOK = 262144
+
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+
+models = client.models.list()
+model = models.data[0].id
+
+
+def load_system_prompt(repo_id: str, filename: str) -> str:
+    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
+    with open(file_path, "r") as file:
+        system_prompt = file.read()
+    today = datetime.today().strftime("%Y-%m-%d")
+    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
+    model_name = repo_id.split("/")[-1]
+    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)
+
+
+SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")
+image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438"
+
+messages = [
+    {"role": "system", "content": SYSTEM_PROMPT},
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "text",
+                "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.",
+            },
+            {"type": "image_url", "image_url": {"url": image_url}},
+        ],
+    },
+]
+
+print(messages)
+
+
+response = client.chat.completions.create(
+    model=model,
+    messages=messages,
+    temperature=TEMP,
+    max_tokens=MAX_TOK,
+)
+
+print(response.choices[0].message.content)
+```
+
+</details>
+
+<details>
+  <summary>Function Calling</summary>
+
+Let's solve some equations thanks to our simple Python calculator tool.
+
+```python
+import json
+from openai import OpenAI
+from huggingface_hub import hf_hub_download
+
+# Modify OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "EMPTY"
+openai_api_base = "http://localhost:8000/v1"
+
+TEMP = 0.15
+MAX_TOK = 262144
+
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+
+models = client.models.list()
+model = models.data[0].id
+
+
+def load_system_prompt(repo_id: str, filename: str) -> str:
+    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
+    with open(file_path, "r") as file:
+        system_prompt = file.read()
+    return system_prompt
+
+
+SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")
+
+image_url = "https://math-coaching.com/img/fiche/46/expressions-mathematiques.jpg"
+
+
+def my_calculator(expression: str) -> str:
+    return str(eval(expression))
+
+
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "my_calculator",
+            "description": "A calculator that can evaluate a mathematical expression.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "expression": {
+                        "type": "string",
+                        "description": "The mathematical expression to evaluate.",
+                    },
+                },
+                "required": ["expression"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "rewrite",
+            "description": "Rewrite a given text for improved clarity",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "text": {
+                        "type": "string",
+                        "description": "The input text to rewrite",
+                    }
+                },
+            },
+        },
+    },
+]
+
+messages = [
+    {"role": "system", "content": SYSTEM_PROMPT},
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "text",
+                "text": "Thanks to your calculator, compute the results for the equations that involve numbers displayed in the image.",
+            },
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": image_url,
+                },
+            },
+        ],
+    },
+]
+
+response = client.chat.completions.create(
+    model=model,
+    messages=messages,
+    temperature=TEMP,
+    max_tokens=MAX_TOK,
+    tools=tools,
+    tool_choice="auto",
+)
+
+tool_calls = response.choices[0].message.tool_calls
+
+results = []
+for tool_call in tool_calls:
+    function_name = tool_call.function.name
+    function_args = tool_call.function.arguments
+    if function_name == "my_calculator":
+        result = my_calculator(**json.loads(function_args))
+        results.append(result)
+
+messages.append({"role": "assistant", "tool_calls": tool_calls})
+for tool_call, result in zip(tool_calls, results):
+    messages.append(
+        {
+            "role": "tool",
+            "tool_call_id": tool_call.id,
+            "name": tool_call.function.name,
+            "content": result,
+        }
+    )
+
+
+response = client.chat.completions.create(
+    model=model,
+    messages=messages,
+    temperature=TEMP,
+    max_tokens=MAX_TOK,
+)
+
+print(response.choices[0].message.content)
+```
+
+</details>
+
+<details>
+  <summary>Text-Only Request</summary>
+
+Ministral 3 can follow your instructions to the letter.
+
+```python
+from openai import OpenAI
+from huggingface_hub import hf_hub_download
+
+# Modify OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "EMPTY"
+openai_api_base = "http://localhost:8000/v1"
+
+TEMP = 0.15
+MAX_TOK = 262144
+
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+
+models = client.models.list()
+model = models.data[0].id
+
+
+def load_system_prompt(repo_id: str, filename: str) -> str:
+    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
+    with open(file_path, "r") as file:
+        system_prompt = file.read()
+    return system_prompt
+
+
+SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")
+
+messages = [
+    {"role": "system", "content": SYSTEM_PROMPT},
+    {
+        "role": "user",
+        "content": "Write me a sentence where every word starts with the next letter in the alphabet - start with 'a' and end with 'z'.",
+    },
+]
+
+response = client.chat.completions.create(
+    model=model,
+    messages=messages,
+    temperature=TEMP,
+    max_tokens=MAX_TOK,
+)
+
+assistant_message = response.choices[0].message.content
+print(assistant_message)
+```
+
+</details>
+
+### Transformers
+
+You can also use Ministral 3 3B Instruct 2512 with `Transformers` !
+
+Transformers very recently added prelimenary support for FP8, so please make sure to install from main:
+
+```sh
+uv pip install git+https://github.com/huggingface/transformers
+```
+
+To make the best use of our model with `Transformers` make sure to have [installed](https://github.com/mistralai/mistral-common) `mistral-common >= 1.8.6` to use our tokenizer.
+
+```bash
+pip install mistral-common --upgrade
+```
+
+Try it out by running the following snippet.
+
+> [!Tip]
+> By default Transformers will load the checkpoint in FP8 and dequantize it to BF16 on the fly,
+> which means the model currently does not make use of accelerated FP8-kernels.
+> Compatibility with accelerated FP8-kernels is currently worked on and will be available in a couple of weeks.
+> Stay tuned!
+
+<details>
+  <summary>Python snippet</summary>
+
+```python
+import torch
+from transformers import Mistral3ForConditionalGeneration, MistralCommonBackend
+
+model_id = "mistralai/Ministral-3-3B-Instruct-2512"
+
+tokenizer = MistralCommonBackend.from_pretrained(model_id)
+model = Mistral3ForConditionalGeneration.from_pretrained(model_id, device_map="auto")
+
+image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438"
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "text",
+                "text": "What action do you think I should take in this situation? List all the possible actions and explain why you think they are good or bad.",
+            },
+            {"type": "image_url", "image_url": {"url": image_url}},
+        ],
+    },
+]
+
+tokenized = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True)
+
+tokenized["input_ids"] = tokenized["input_ids"].to(device="cuda")
+tokenized["pixel_values"] = tokenized["pixel_values"].to(dtype=torch.bfloat16, device="cuda")
+image_sizes = [tokenized["pixel_values"].shape[-2:]]
+
+output = model.generate(
+    **tokenized,
+    image_sizes=image_sizes,
+    max_new_tokens=512,
+)[0]
+
+decoded_output = tokenizer.decode(output[len(tokenized["input_ids"][0]):])
+print(decoded_output)
+```
+
+**Note:**
+
+Transformers allows you to automatically convert the checkpoint to Bfloat16. To so simple load the model as follows:
+
+```py
+from transformers import Mistral3ForConditionalGeneration, FineGrainedFP8Config
+
+model_id = "mistralai/Ministral-3-3B-Instruct-2512"
+model = Mistral3ForConditionalGeneration.from_pretrained(
+    model_id,
+    device_map="auto",
+    quantization_config=FineGrainedFP8Config(dequantize=True)
+)
+```
+
+</details>
+
 ## License
 
 This model is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0.txt).