Update README.md (#31)
Browse files- Update README.md (c3e22d3f9f15649c42fb3d61ed29e609d85d0339)
README.md
CHANGED
|
@@ -126,7 +126,11 @@ We recommend that you use Mistral-Small-3.2-24B-Instruct-2506 in a server/client
|
|
| 126 |
1. Spin up a server:
|
| 127 |
|
| 128 |
```
|
| 129 |
-
vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
```
|
| 131 |
|
| 132 |
**Note:** Running Mistral-Small-3.2-24B-Instruct-2506 on GPU requires ~55 GB of GPU RAM in bf16 or fp16.
|
|
|
|
| 126 |
1. Spin up a server:
|
| 127 |
|
| 128 |
```
|
| 129 |
+
vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506 \
|
| 130 |
+
--tokenizer_mode mistral --config_format mistral \
|
| 131 |
+
--load_format mistral --tool-call-parser mistral \
|
| 132 |
+
--enable-auto-tool-choice --limit-mm-per-prompt '{"image":10}' \
|
| 133 |
+
--tensor-parallel-size 2
|
| 134 |
```
|
| 135 |
|
| 136 |
**Note:** Running Mistral-Small-3.2-24B-Instruct-2506 on GPU requires ~55 GB of GPU RAM in bf16 or fp16.
|