have some questions:

1. model has unused tensor blk.92 -- ignoring ?
   is this correct?
2. How can I increase the context size, the maximum for my workstation is less than 60K

Hmm I want to know since the QYFS + 512 GB DDR5 + Sage SE combo...

Have you tried Deepseek (latest of V3 or R1) I want to know the token gen speeds you get from say a IQ4 quant 32k Context?

DeepSeek-V3-0324-IQ4_K_R4

main: n_kv_max = 65536, n_batch = 4090, n_ubatch = 4090, flash_attn = 1, n_gpu_layers = 63, n_threads = 101, n_threads_batch = 101

DeepSeek-R1-0528-IQ4_KS_R4

main: n_kv_max = 65536, n_batch = 4090, n_ubatch = 4090, flash_attn = 1, n_gpu_layers = 63, n_threads = 101, n_threads_batch = 101

GLM 4.5 IQ4_K

Tensor blk.91.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.91.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.91.ffn_up_exps.weight buffer type overriden to CPU
model has unused tensor blk.92.attn_norm.weight (size = 20480 bytes) -- ignoring
model has unused tensor blk.92.attn_q.weight (size = 52101120 bytes) -- ignoring
model has unused tensor blk.92.attn_k.weight (size = 5570560 bytes) -- ignoring
model has unused tensor blk.92.attn_v.weight (size = 5570560 bytes) -- ignoring
model has unused tensor blk.92.attn_q.bias (size = 49152 bytes) -- ignoring
model has unused tensor blk.92.attn_k.bias (size = 4096 bytes) -- ignoring
model has unused tensor blk.92.attn_v.bias (size = 4096 bytes) -- ignoring
model has unused tensor blk.92.attn_output.weight (size = 52101120 bytes) -- ignoring
model has unused tensor blk.92.attn_q_norm.weight (size = 512 bytes) -- ignoring
model has unused tensor blk.92.attn_k_norm.weight (size = 512 bytes) -- ignoring
model has unused tensor blk.92.post_attention_norm.weight (size = 20480 bytes) -- ignoring
model has unused tensor blk.92.ffn_gate_inp.weight (size = 3276800 bytes) -- ignoring
model has unused tensor blk.92.exp_probs_b.bias (size = 640 bytes) -- ignoring
Tensor blk.92.ffn_gate_exps.weight buffer type overriden to CPU
model has unused tensor blk.92.ffn_gate_exps.weight (size = 707788800 bytes) -- ignoring
Tensor blk.92.ffn_down_exps.weight buffer type overriden to CPU
model has unused tensor blk.92.ffn_down_exps.weight (size = 865075200 bytes) -- ignoring
Tensor blk.92.ffn_up_exps.weight buffer type overriden to CPU
model has unused tensor blk.92.ffn_up_exps.weight (size = 707788800 bytes) -- ignoring
model has unused tensor blk.92.ffn_gate_shexp.weight (size = 6512640 bytes) -- ignoring
model has unused tensor blk.92.ffn_down_shexp.weight (size = 8355840 bytes) -- ignoring
model has unused tensor blk.92.ffn_up_shexp.weight (size = 6512640 bytes) -- ignoring
model has unused tensor blk.92.nextn.eh_proj.weight (size = 55705600 bytes) -- ignoring
model has unused tensor blk.92.nextn.embed_tokens.weight (size = 533463040 bytes) -- ignoring
model has unused tensor blk.92.nextn.enorm.weight (size = 20480 bytes) -- ignoring
model has unused tensor blk.92.nextn.hnorm.weight (size = 20480 bytes) -- ignoring
model has unused tensor blk.92.nextn.shared_head_head.weight (size = 533463040 bytes) -- ignoring
model has unused tensor blk.92.nextn.shared_head_norm.weight (size = 20480 bytes) -- ignoring
llm_load_tensors: offloading 93 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 94/94 layers to GPU
llm_load_tensors: CPU buffer size = 40746.39 MiB
llm_load_tensors: CPU buffer size = 42230.00 MiB
llm_load_tensors: CPU buffer size = 42380.00 MiB
llm_load_tensors: CPU buffer size = 42905.00 MiB
llm_load_tensors: CPU buffer size = 37416.98 MiB
llm_load_tensors: CPU buffer size = 416.25 MiB
llm_load_tensors: CUDA0 buffer size = 13323.87 MiB
....................................................................................................

MLA is only available for LLM_ARCH_DEEPSEEK2 -> turning off MLA

llama_new_context_with_model: n_ctx = 68352
llama_new_context_with_model: n_batch = 4096
llama_new_context_with_model: n_ubatch = 4096
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: mla_attn = 0
llama_new_context_with_model: attn_max_b = 512
llama_new_context_with_model: fused_moe = 1
llama_new_context_with_model: ser = -1, 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 13191.51 MiB
llama_new_context_with_model: KV self size = 13191.47 MiB, K (q8_0): 6595.73 MiB, V (q8_0): 6595.73 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.58 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 2978.02 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 1148.05 MiB
llama_new_context_with_model: graph nodes = 4457
llama_new_context_with_model: graph splits = 180
XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MUL_MAT_ID to OFF
XXXXXXXXXXXXXXXXXXXXX Setting offload policy for op MOE_FUSED_UP_GATE to OFF

main: n_kv_max = 68352, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, n_gpu_layers = 99, n_threads = 101, n_threads_batch = 101

hmm....
I thought the GLM being relatively less in size would make it have more TG speed than the Deepseeks

@shewin

model has unused tensor blk.92 -- ignoring ? is this correct?

Yes, for both GLM models the last layer is set with the new TENSOR_SKIP flag and is not involved with inference. It only takes up space on the disk, and is not loaded into VRAM/RAM. This layer has the NextN MTP tensors which are not yet used anywhere but may become used in the future.

How can I increase the context size, the maximum for my workstation is less than 60K

You can use -ctk q6_0 -ctv q6_0 to free up some more VRAM to fit more context for example, or offload less extra exps layer to VRAM etc.

@SFPLM

hmm.... I thought the GLM being relatively less in size would make it have more TG speed than the Deepseeks

So DeepSeek is about 37B active parameters and GLM-4.5 is about 32B active parameters for TG so it should be slightly faster assuming memory bandwidth is the limit and similar quantization size.

I'd have to see the full command from @shewin including how many GPUs, system CPU and RAM speed, and Linux or Windows to fully know. In general I no longer use the -rtr and instead omitt it and use -ub 4096 -b 4096 which can really boost PP speed. However leaving it at default batch sizes and going with -rtr and using the extra VRAM to offload a few more exps layers may improve TG on GLM-4.5 but I have not tested.

with -ctk q6_0 -ctv q6_0
context size 10 - 20K increased

IQ5_K:
main: n_kv_max = 71168, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, n_gpu_layers = 99, n_threads = 101, n_threads_batch = 101

IQ4_K:
main: n_kv_max = 90112, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, n_gpu_layers = 99, n_threads = 101, n_threads_batch = 101

main: n_kv_max = 131072, n_batch = 8192, n_ubatch = 8192, flash_attn = 1, n_gpu_layers = 999, n_threads = 57, n_threads_batch = 57

I also get similar numbers with similar config but using anikifoss' HQ4 Quant. I decided to end there because I did not want to wait.

@ubergarm
Do you know if MLA has potential to be supported for GLM models? does this play a role in the impact of TG speeds?

@SFPLM

n_batch = 8192, n_ubatch = 8192

Some folks have reported success with the larger batch sizes, but some others have had issues going higher. I tend to stick to 4096 or below usually but looks like you got it working there!

Do you know if MLA has potential to be supported for GLM models? does this play a role in the impact of TG speeds?

MLA would be multi-head latent attention similar to deepseek and kimi-k2. there are some papers discussing implementing MLA instead of say GQA etc on existing architectures but it would take more fine-tuning at least to get it working so beyond the scope of most of us home users to adapt it ourselves. MLA has both pros and cons in that it can reduce the size of the kv-cache (it grows linearly instead of exponentially) however it takes more computation to compress/decompress it to/from latent space so not necessarily faster.