Upload CosmosGemma T1 GGUF models

.gitattributes

@@ -33,3 +33,26 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1-F16.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q3_K.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q4_K.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q5_1.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q5_K.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Turkish-Gemma-9b-T1.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,99 @@
+---
+base_model: ytu-ce-cosmos/Turkish-Gemma-9b-T1
+license: gemma
+language:
+- tr
+- en
+tags:
+- gguf
+- ggml
+- gemma
+- cosmosgemma
+- turkish gemma
+- t1
+---
+# CosmosGemma T1 GGUFs
+
+## Objective
+Due to the need for quantized models in real-time applications, we introduce our GGUF formatted models. These models are part of 
+GGML project with a hope to democratize the use of Large Models. Depending on the quantization type, there are 20+ models.
+
+### Features
+* All quantization details are listed on the right by Hugging Face.
+* All the models have been tested in `llama.cpp` environments, `llama-cli` and `llama-server`. 
+* Furthermore, a YouTube video has been made to introduce the basics of using `lmstudio` to utilize these models. 👇
+[![lmstudio_yt](https://img.youtube.com/vi/JRID-6sRl7I/0.jpg)](https://www.youtube.com/watch?v=JRID-6sRl7I)
+
+
+### Code Example
+Usage example with `llama-cpp-python`
+
+```py
+from llama_cpp import Llama
+
+# Define the inference parameters
+inference_params = {
+    "n_threads": 4,
+    "n_predict": -1,
+    "top_k": 40,
+    "min_p": 0.05,
+    "top_p": 0.95,
+    "temp": 0.8,
+    "repeat_penalty": 1.1,
+    "input_prefix": "<start_of_turn>user\\n",
+    "input_suffix": "<end_of_turn>\\n<start_of_turn>model\\n",
+    "antiprompt": [],
+    "pre_prompt": "Sen bir yapay zeka asistanısın. Kullanıcı sana bir görev verecek. Amacın görevi olabildiğince sadık bir şekilde tamamlamak.",
+    "pre_prompt_suffix": "<end_of_turn>\\n",
+    "pre_prompt_prefix": "<bos><start_of_turn>system\\n",
+    "seed": -1,
+    "tfs_z": 1,
+    "typical_p": 1,
+    "repeat_last_n": 64,
+    "frequency_penalty": 0,
+    "presence_penalty": 0,
+    "n_keep": 0,
+    "logit_bias": {},
+    "mirostat": 0,
+    "mirostat_tau": 5,
+    "mirostat_eta": 0.1,
+    "memory_f16": True,
+    "multiline_input": False,
+    "penalize_nl": True
+}
+
+# Initialize the Gemma model with the specified inference parameters
+gemma = Llama.from_pretrained(
+    repo_id="ytu-ce-cosmos/Turkish-Gemma-9b-T1-GGUF",
+    filename="*Q4_K.gguf",
+    verbose=False
+)
+# Example input
+user_input = "Türkiyenin başkenti neresidir?"
+
+# Construct the prompt
+prompt = f"{inference_params['pre_prompt_prefix']}{inference_params['pre_prompt']}{inference_params['pre_prompt_suffix']}{inference_params['input_prefix']}{user_input}{inference_params['input_suffix']}"
+
+# Generate the response
+response = gemma(prompt)
+
+# Output the response
+print(response['choices'][0]['text'])
+
+```
+
+The quantization has been made using `llama.cpp`. As we have seen, this method tends to give the most stable results.
+
+Obviously, we encountered better inference quality for models with the highest bits. However, the inference time tends to be similar between low-bit models. 
+
+Each model's memory footprint can be anticipated by the qunatization docs in either [Hugging Face](https://huggingface.co/docs/transformers/main/en/quantization/overview) or [llama.cpp](https://github.com/ggerganov/llama.cpp/tree/master/examples/quantize).
+
+
+# Acknowledgments
+- Research supported with Cloud TPUs from [Google's TensorFlow Research Cloud](https://sites.research.google/trc/about/) (TFRC). Thanks for providing access to the TFRC ❤️
+- Thanks to the generous support from the Hugging Face team, it is possible to download models from their S3 storage 🤗
+
+## Contact
+COSMOS AI Research Group, Yildiz Technical University Computer Engineering Department  
+https://cosmos.yildiz.edu.tr/  
+[email protected]

Turkish-Gemma-9b-T1-F16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f1602d5c860a6ade3e290a257b4fc262432155bf84b1aad0d195a1b2540a19d
+size 18490680320

Turkish-Gemma-9b-T1.IQ2_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c2e67de6ff7981ddd82f48fd01ebf41137bca0057feb8b4c20b7304b307f0b9
+size 636855296

Turkish-Gemma-9b-T1.IQ3_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:861508d08584373d8270714dc3c8ddf3493e51d4eba28e7b056ec19c5d8ace6b
+size 4494615552

Turkish-Gemma-9b-T1.IQ3_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:430e0cb276349f0a38a402c44cd2d6541412f662a0531dc3125ea18278600243
+size 4337665024

Turkish-Gemma-9b-T1.IQ3_XS.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea2e7477661a131914df70bf66c44aa09ea3294fbc1c7019cc33e27254566944
+size 4144989184

Turkish-Gemma-9b-T1.IQ3_XXS.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c2e67de6ff7981ddd82f48fd01ebf41137bca0057feb8b4c20b7304b307f0b9
+size 636855296

Turkish-Gemma-9b-T1.IQ4_NL.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:51a750b61f5470205851d34e8c7c354c0ef148a4258ffd535c715557171595d1
+size 5475255296

Turkish-Gemma-9b-T1.IQ4_XS.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e199c0abeaa98ad344ec7120391447f5eb2c94e743019e119d6d712df4e0ae5
+size 5223171072

Turkish-Gemma-9b-T1.Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:182c70eac8c84228b8d1d7a5cbdead348f4f13dd2cdaf2df6352b87a8599f0c9
+size 3805398016

Turkish-Gemma-9b-T1.Q3_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38b5513c53e46575cb70fb16cec16a5e07864447a4e2fe779ce19c54e2642f73
+size 4761781248

Turkish-Gemma-9b-T1.Q3_K_L.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff59485dbd8065653a5790248af166619e062e119b59b41fc5689b9c64f3836a
+size 5132452864

Turkish-Gemma-9b-T1.Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38b5513c53e46575cb70fb16cec16a5e07864447a4e2fe779ce19c54e2642f73
+size 4761781248

Turkish-Gemma-9b-T1.Q3_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d43aefa64ab30070fbe74816f30563fb7e04f65fa82501c6a64336d87f5be3bf
+size 4337665024

Turkish-Gemma-9b-T1.Q4_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e49387c2d8ab1d1a4d65f3e720e8fef190a87e6049b424369779847ee2832c43
+size 5761057792

Turkish-Gemma-9b-T1.Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e49387c2d8ab1d1a4d65f3e720e8fef190a87e6049b424369779847ee2832c43
+size 5761057792

Turkish-Gemma-9b-T1.Q4_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e6776dc02bf5fb07b40b6e948affdc3f979286b3e6b3d02959306060a907d71
+size 5478925312

Turkish-Gemma-9b-T1.Q5_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3c0723e1c323519684bc62378da1b340a08cf16e17054830f9e0ae79f511651
+size 6483592192

Turkish-Gemma-9b-T1.Q5_1.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fe79fcbe9d8c9cd826c33e738a093dfb7458bb6347cb084e01f6dbc4fa7efdb
+size 7003816960

Turkish-Gemma-9b-T1.Q5_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61f5743d26e6f826aa0c494dcf198d6944832b251b5c52f8f80efea36423a92b
+size 6647366656

Turkish-Gemma-9b-T1.Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61f5743d26e6f826aa0c494dcf198d6944832b251b5c52f8f80efea36423a92b
+size 6647366656

Turkish-Gemma-9b-T1.Q5_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b23a9f1dbd3511802a7e25d03789df56d83c221f1d0a302b8e7afea0380d751
+size 6483592192

Turkish-Gemma-9b-T1.Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:adfb8160fb498505730f8465da99634328d7a7640776c63cf5540738607d147b
+size 7589069824

Turkish-Gemma-9b-T1.Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f23708d2eeee52229027a2da3ebf2974aa89012f48674f5cc602148860ee4e2
+size 9827148800

cosmosgemma_t1_lm_studio_preset.json

@@ -0,0 +1,49 @@
+{
+  "name": "cosmosgemma_t1_lm_studio",
+  "load_params": {
+    "n_ctx": 2048,
+    "n_batch": 512,
+    "rope_freq_base": 0,
+    "rope_freq_scale": 0,
+    "n_gpu_layers": 10,
+    "use_mlock": true,
+    "main_gpu": 0,
+    "tensor_split": [
+      0
+    ],
+    "seed": -1,
+    "f16_kv": true,
+    "use_mmap": true,
+    "no_kv_offload": false,
+    "num_experts_used": 0
+  },
+  "inference_params": {
+    "n_threads": 4,
+    "n_predict": -1,
+    "top_k": 40,
+    "min_p": 0.05,
+    "top_p": 0.95,
+    "temp": 0.8,
+    "repeat_penalty": 1.1,
+    "input_prefix": "<start_of_turn>user\\n",
+    "input_suffix": "<end_of_turn>\\n<start_of_turn>model\\n",
+    "antiprompt": [],
+    "pre_prompt": "Sen bir yapay zeka asistanısın. Kullanıcı sana bir görev verecek. Amacın görevi olabildiğince sadık bir şekilde tamamlamak. Görevi yerine getirirken adım adım düşün ve adımlarını gerekçelendir.",
+    "pre_prompt_suffix": "<end_of_turn>\\n",
+    "pre_prompt_prefix": "<bos><start_of_turn>system\\n",
+    "seed": -1,
+    "tfs_z": 1,
+    "typical_p": 1,
+    "repeat_last_n": 64,
+    "frequency_penalty": 0,
+    "presence_penalty": 0,
+    "n_keep": 0,
+    "logit_bias": {},
+    "mirostat": 0,
+    "mirostat_tau": 5,
+    "mirostat_eta": 0.1,
+    "memory_f16": true,
+    "multiline_input": false,
+    "penalize_nl": true
+  }
+}