8bit AWQ
#2
by
MrMoonsilver
- opened
Hey, first of all thank you so much for your work quantizing to AWQ all these models. Really appreciate it!
Do you plan to release an 8bit version of this? Also, have you plans to quantize the new magistral 24b model too?
Hi @MrMoonsilver , I am happy that you are interested in my models!
I do plan to release an 8bit version in the next few days, as I've just completed finalizing the 4bit model.
I also already completed quantizing the new magistral 24b, but I am trying to convert into mistral configs and formats for vllm to work.
Thank you, so much, your quantizations are the best :]