Create README.md

README.md

@@ -0,0 +1,20 @@
+---
+base_model:
+- meta-llama/Meta-Llama-3.1-8B-Instruct
+---
+
+
+# A 4bit AWQ version of meta-llama/Meta-Llama-3.1-8B-Instruct for the LMDeploy TurboMindEngine
+
+```
+lmdeploy lite auto_awq \
+   $HF_MODEL \
+  --calib-dataset 'ptb' \
+  --calib-samples 128 \
+  --calib-seqlen 2048 \
+  --w-bits 4 \
+  --w-group-size 128 \
+  --batch-size 10 \
+  --search-scale True \
+  --work-dir $WORK_DIR
+```