Need GGUF Quantization
#2
by tradenity - opened
Please provide GGUF quantization like the 7B model in order to be able to use it with ollama
It would be cool if the unsloth team released a GGUF version.
llama.cpp must be supported first. Then gguf
q8 pls
Team mradermacher agreed to quantize it, but it failed the pipeline as of now: https://huggingface.co/mradermacher/model_requests/discussions/2423
great
I made a q4 for testing. feel free to check it out