llama.cpp
Can you add support for llama.cpp ?
This is work in progress, we will post an update here once there is a GGUF or llama.cpp pull request available.
Much obliged
I all! llama.cpp support is partially ready: https://github.com/ggml-org/llama.cpp/pull/16112
There appears to be some disconnect between the vision encoder logic in mtmd and the logic for idefics3 in transormers, so the results I'm seeing are quite poor currently. I plan to work on that in a separate PR soon.
Work on this has moved to a new PR since the llama.cpp team did a reshuffle of roles recently: https://github.com/ggml-org/llama.cpp/pull/16206
NOTE: The PR now contains the addition of a new GGUF metadata field which could change based on review comments, so GGUFs produced using this branch will not be guaranteed to work in the long run until review is finished.