[🏠Sentient Simulations] | [Discord] | [Patreon]

---

BlackSheep-24B-GPTQ

This repository contains a 4 bit GPTQ-quantized version of the TroyDoesAI/BlackSheep-24B model using llm-compressor.

Quantization Settings

Attribute	Value
Algorithm	GPTQ
Layers	Linear
Weight Scheme	W4A16
Group Size	128
Calibration Dataset	openerotica/erotiquant3
Calibration Sequence Length	4900
Calibration Samples	1024

Dataset Preprocessing

The dataset was preprocessed with the following steps:

1. Extract and structure the conversation data using role-based templates (SYSTEM, USER, ASSISTANT).
2. Convert the structured conversations into a tokenized format using the model's tokenizer.
3. Filter out sequences shorter than 4096 tokens.
4. Shuffle and select 512 samples for calibration.

Quantization Process

View the shell and python script used to quantize this model.

1 local 3090 128GB of ram.

• compress.sh
• compress.py

Acknowledgments

• Base Model: TroyDoesAI/BlackSheep-24B
• Calibration Dataset: openerotica/erotiquant3
• LLM Compressor: llm-compressor
• Everyone subscribed to the Sentient Simulations Patreon