| Quantized : https://huggingface.co/Sao10K/Euryale-L2-70B | |
| With : https://github.com/turboderp/exllamav2 | |
| Fits into 24GB of ram with 4096 context. Unfortunately it seems to be dumbed down a bit too much by compression. Files also include measurement.json that can be used to speed up quantization process for other BPW size. | |
| Measurement done with default parameters and https://huggingface.co/datasets/wikitext/tree/refs%2Fconvert%2Fparquet/wikitext-103-raw-v1/test | |
| --- | |
| license: other | |
| --- | |