Tool call parsing error
#10
by
wasiuddina
- opened
vllm/vllm-openai:nightly is giving error.
(APIServer pid=37646) INFO 12-11 09:36:12 [chat_utils.py:574] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=37646) INFO 12-11 09:36:26 [loggers.py:248] Engine 000: Avg prompt throughput: 36269.9 tokens/s, Avg generation throughput: 66.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 82.1%, MM cache hit rate: 0.0%
(APIServer pid=37646) INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=37646) WARNING 12-11 09:36:28 [mistral.py:77] Truncating tool call ID: chatcmpl-tool-8d850927cf39be3c to 7cf39be3c
(APIServer pid=37646) WARNING 12-11 09:36:28 [mistral.py:91] Truncating tool_call_id: chatcmpl-tool-8d850927cf39be3c to 7cf39be3c
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] An error occurred in `mistral_common` while applying chat template
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] Traceback (most recent call last):
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 1811, in apply_mistral_chat_template
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] return tokenizer.apply_chat_template(
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/mistral.py", line 436, in apply_chat_template
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] return self.transformers_tokenizer.apply_chat_template(
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_mistral_common.py", line 1504, in apply_chat_template
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] chat_request = ChatCompletionRequest.from_openai(
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] File "/usr/local/lib/python3.12/dist-packages/mistral_common/protocol/instruct/request.py", line 185, in from_openai
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] converted_messages: list[ChatMessage] = convert_openai_messages(messages)
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] File "/usr/local/lib/python3.12/dist-packages/mistral_common/protocol/instruct/converters.py", line 33, in convert_openai_messages
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] message = ToolMessage.from_openai(openai_message)
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] File "/usr/local/lib/python3.12/dist-packages/mistral_common/protocol/instruct/messages.py", line 287, in from_openai
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] tool_message = cls.model_validate(messages)
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] File "/usr/local/lib/python3.12/dist-packages/pydantic/main.py", line 716, in model_validate
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] return cls.__pydantic_validator__.validate_python(
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] pydantic_core._pydantic_core.ValidationError: 1 validation error for ToolMessage
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] content
(APIServer pid=37646) ERROR 12-11 09:36:28 [chat_utils.py:1828] Input should be a valid string [type=string_type, input_value=[{'text': 'Your thought h...gged.', 'type': 'text'}], input_type=list]
Which vllm container I should use? Can someone help?
Hmm.. I don't think tool message content in the format: Which client/scaffold are you making requests with?[{'text': 'Your thought h...gged.', 'type': 'text'}] is a standard openai format.
Edit: it does seem to be part of the standard api spec: https://github.com/openai/openai-openapi/blob/manual_spec/openapi.yaml#L18184.
We use openai client and openhands as scaffold.
