bartowski

/

ai21labs_AI21-Jamba-Reasoning-3B-GGUF

like 5

The q4km from ai21labs themselves works

https://huggingface.co/ai21labs/AI21-Jamba-Reasoning-3B-GGUF/blob/main/jamba-reasoning-3b-Q4_K_M.gguf

@AaronFeng753 thanks for pointing it out, there was a tokenizer issue that I think i've fixed now (tested the bf16 locally and it worked), can you give it another try?

Also seems to work fine in lmstudio now (though thinking tags aren't being properly accounted for at this time somehow)

Also seems to work fine in lmstudio now (though thinking tags aren't being properly accounted for at this time somehow)

Hi, I've noticed the missing think tags in LM Studio using the official F16 GGUF. It is weird seeing how it worked well for @AaronFeng753 in OpenWebUI? Is it using different backend than LM Studio, maybe? But I guess this is connected through the API, maybe it works in server mode, but not in the LM Studio UI directly?

The model itself works fine, but something seems still wrong with the jinja template. This happens when I try to use the model with tools:

500: 500 Trying to call method 'startswith' on null at row 37, column 84:
            {%- if not loop.last %}
                {%- if loop.nextitem.role == 'assistant' and loop.nextitem.content.startswith('<think>') or loop.nextitem.reasoning_content is defined and loop.nextitem.reasoning_content is not none %}
                                                                                   ^
[...]

I am using this template without issues:

{%- set thinking_prefix = thinking_prefix or 'Begin by thinking about the reasoning process in the mind within <think> </think> tags and then proceed to give your response.\n' -%} {%- if bos_token is defined and bos_token is not none %}{{- bos_token -}}{%- endif %} {%- if tools %} {{- '<|im_start|>system\n' }} {%- if messages|length > 0 and messages[0].role == 'system' %} {{- messages[0].content + '\n\n' }} {%- endif %} {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }} {%- for tool in tools %} {{- "\n" }} {{- tool | tojson }} {%- endfor %} {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }} {%- else %} {%- if messages|length > 0 and messages[0].role == 'system' %} {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} {%- for message in messages[::-1] %} {%- set index = (messages|length - 1) - loop.index0 %} {%- if ns.multi_step_tool and message.role == "user" and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %} {%- set ns.multi_step_tool = false %} {%- set ns.last_query_index = index %} {%- endif %} {%- endfor %} {%- for message in messages %} {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} {%- set prefix = '' %} {%- if message.role == 'user' and '<think>' not in message.content %} {%- if loop.last %} {%- set prefix = thinking_prefix %} {%- endif %} {%- if not loop.last %} {%- if loop.nextitem.role == 'assistant' and (loop.nextitem.content is not none and loop.nextitem.content.startswith('<think>')) or loop.nextitem.reasoning_content is defined and loop.nextitem.reasoning_content is not none %} {%- set prefix = thinking_prefix %} {%- endif %} {%- endif %} {%- endif %} {%- if message.role == 'user' and not loop.last and loop.nextitem.role == 'assistant' and (loop.nextitem.content is not none and loop.nextitem.content.startswith('<think>')) and '<think>' not in message.content %} {%- set prefix = thinking_prefix %} {%- endif %} {{- '<|im_start|>' + message.role + '\n' + prefix + message.content + '<|im_end|>' + '\n' }} {%- elif message.role == "assistant" %} {%- set content = message.content %} {%- set reasoning_content = '' %} {%- if message.reasoning_content is defined and message.reasoning_content is not none %} {%- set reasoning_content = message.reasoning_content %} {%- else %} {%- if '</think>' in message.content %} {%- set content = message.content.split('</think>')[-1].lstrip('\n') %} {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %} {%- endif %} {%- endif %} {%- if loop.index0 > ns.last_query_index %} {%- if reasoning_content %} {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }} {%- else %} {{- '<|im_start|>' + message.role + '\n' + content }} {%- endif %} {%- else %} {{- '<|im_start|>' + message.role + '\n' + content }} {%- endif %} {%- if message.tool_calls %} {%- for tool_call in message.tool_calls %} {%- if (loop.first and content) or (not loop.first) %} {{- '\n' }} {%- endif %} {%- if tool_call.function %} {%- set tool_call = tool_call.function %} {%- endif %} {{- '<tool_call>\n{"name": "' }} {{- tool_call.name }} {{- '", "arguments": ' }} {%- if tool_call.arguments is string %} {{- tool_call.arguments }} {%- else %} {{- tool_call.arguments | tojson }} {%- endif %} {{- '}\n</tool_call>' }} {%- endfor %} {%- endif %} {{- '<|im_end|>\n' }} {%- elif message.role == "tool" %} {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} {{- '<|im_start|>user' }} {%- endif %} {{- '\n<tool_response>\n' }} {{- message.content }} {{- '\n</tool_response>' }} {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} {{- '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- endfor %} {%- if add_generation_prompt %} {{- '<|im_start|>assistant\n' }} {{- '<think>\n' }} {%- endif -%}