Hugging Face's logo

EVA-UNIT-01
/
EVA-Qwen2.5-72B-v0.0

Text Generation
Transformers
Safetensors
qwen2
Generated from Trainer
conversational
text-generation-inference
Model card Files
xet
Community
3
lbourdois commited on
Commit
868fd5d
·
verified ·
1 Parent(s): 611a9d3

Improve language tag

Browse files

Hi! As the model is multilingual, this is a PR to add other languages than English to the language tag to improve the referencing. Note that 29 languages are announced in the README, but only 13 are explicitly listed. I was therefore only able to add these 13 languages.

Files changed (1) hide show
  1. README.md +498 -484
README.md CHANGED
@@ -1,485 +1,499 @@
1
- ---
2
- library_name: transformers
3
- license: other
4
- license_name: qwen
5
- license_link: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE
6
- base_model: Qwen/Qwen2.5-72B
7
- datasets:
8
- - anthracite-org/kalo-opus-instruct-22k-no-refusal
9
- - Nopm/Opus_WritingStruct
10
- - Gryphe/Sonnet3.5-SlimOrcaDedupCleaned
11
- - Gryphe/Sonnet3.5-Charcard-Roleplay
12
- - Gryphe/ChatGPT-4o-Writing-Prompts
13
- - Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned
14
- - Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned
15
- - nothingiisreal/Reddit-Dirty-And-WritingPrompts
16
- - allura-org/Celeste-1.x-data-mixture
17
- tags:
18
- - generated_from_trainer
19
- model-index:
20
- - name: EVA-Qwen2.5-72B-SFFT-v0.0
21
- results: []
22
- ---
23
-
24
- # EVA Qwen2.5-72B v0.0
25
-
26
- <p>
27
- A RP/storywriting specialist model, full-parameter finetune of Qwen2.5-72B on mixture of synthetic and natural data.<br>
28
- It uses Celeste 70B 0.1 data mixture, greatly expanding it to improve versatility, creativity and "flavor" of the resulting model.<br>
29
- </p>
30
-
31
- <p>Model is available for inference on <a href=https://featherless.ai/models/EVA-UNIT-01/EVA-Qwen2.5-72B-v0.0>Featherless.AI</a></p
32
-
33
- <p>Note: using quantized KV cache with Qwen2.5 <b>is not recommended</b> and can lead to degraded output quality. On the other hand, Qwen's KV cache is already light enough, so using f16 for it shouldn't be problematic.</p>
34
- <p>Note #2: due to some unexpected effects of data normalization, some artifacting in form of randomly appearring sequence of <code>—</code> can appear in outputs sometimes, if penalties are too high. To avoid it, ban token number <code>158</code>. Thanks to Cahvay/ALK for discovering this fix!</p>
35
- <p>
36
- <p>Prompt format is ChatML.</p><br>
37
- <h3>Recommended sampler values:</h3>
38
- <ul>
39
- <li>Temperature: 1</li>
40
- <li>Typical-P: 0.9</li>
41
- <li>Min-P: 0.05</li>
42
- <li>Top-A: 0.2</li>
43
- <li>Repetition Penalty: 1.03</li>
44
- </ul>
45
-
46
- <h3>Recommended SillyTavern presets (via CalamitousFelicitousness):</h3>
47
-
48
- - [Context](https://huggingface.co/EVA-UNIT-01/EVA-Yi-1.5-9B-32K-V1/blob/main/%5BChatML%5D%20Roleplay-v1.9%20Context.json)
49
- - [Instruct and System Prompt](https://huggingface.co/EVA-UNIT-01/EVA-Yi-1.5-9B-32K-V1/blob/main/%5BChatML%5D%20Roleplay-v1.9%20Instruct.json)
50
- </p>
51
-
52
-
53
- <p>
54
- <br>
55
- <h3>
56
- Training data:
57
- </h3>
58
- <ul>
59
- <li>Celeste 70B 0.1 data mixture minus Opus Instruct subset. See that model's <a href=https://huggingface.co/nothingiisreal/L3.1-70B-Celeste-V0.1-BF16>card</a> for details.</li>
60
- <li>Kalomaze's Opus_Instruct_25k dataset, filtered for refusals.</li>
61
- <li>A subset (1k rows) of ChatGPT-4o-WritingPrompts by Gryphe</li>
62
- <li>A subset (2k rows) of Sonnet3.5-Charcards-Roleplay by Gryphe</li>
63
- <li>Synthstruct and SynthRP datasets by Epiculous</li>
64
- </ul>
65
- <h3>
66
- Training time and hardware:
67
- </h3>
68
- <ul><li>12 hours on 8xMI300X</li></ul><br>
69
- </p>
70
- <p>Model was trained by Kearm and Auri.</p>
71
- <h4>Special thanks:</h4><ul>
72
- <li>to Gryphe, Lemmy, Kalomaze, Nopm and Epiculous for the data</li>
73
- <li>to CalamitiousFelicitousness for providing free inference for public beta testing</li>
74
- <li>and to Allura-org for support and feedback on EVA models.</li></ul>
75
- <a href=https://github.com/axolotl-ai-cloud/axolotl><img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/></a>
76
- <details><summary>See axolotl config</summary>
77
-
78
- axolotl version: `0.4.1`
79
- ```yaml
80
- base_model: Qwen/Qwen2.5-72B
81
-
82
- load_in_8bit: false
83
- load_in_4bit: false
84
- strict: false
85
-
86
- plugins:
87
- - axolotl.integrations.liger.LigerPlugin
88
- liger_rope: true
89
- liger_rms_norm: true
90
- liger_swiglu: false
91
- liger_fused_linear_cross_entropy: false
92
-
93
- # plugins:
94
- # - axolotl.integrations.spectrum.SpectrumPlugin
95
-
96
- # spectrum_top_fraction: 0.5
97
- # # Optional if using a pre-scanned model as your base_model. Useful if using a model mirror
98
- # spectrum_model_name: Qwen/Qwen2.5-32B
99
-
100
- datasets:
101
- - path: datasets/deduped_Synthstruct-Gens_processed_sharegpt_converted_cleaned.jsonl
102
- type: sharegpt
103
- - path: datasets/opus-instruct-22k-no_refusals-filtered.jsonl
104
- type: sharegpt
105
- - path: datasets/Celeste_Filtered.jsonl
106
- type: sharegpt
107
- - path: datasets/Gryphe-S3-5-Charcards-names-2k.jsonl
108
- type: sharegpt
109
- - path: datasets/deduped_SynthRP-Gens_processed_09-25-2024-ShareGPT_converted_cleaned.jsonl
110
- type: sharegpt
111
- - path: datasets/deduped_Gryphe-4o-WP-1k.jsonl
112
- type: sharegpt
113
- - path: datasets/deduped_not_samantha_norefusals.jsonl
114
- type: sharegpt
115
-
116
- chat_template: chatml
117
- shuffle_merged_datasets: true
118
- val_set_size: 0.001
119
- output_dir: ./EVA-Qwen2.5-72B-SFFT-v0.0
120
-
121
- sequence_len: 8192
122
- sample_packing: true
123
- eval_sample_packing: false
124
- pad_to_sequence_len: true
125
-
126
- # adapter: qlora
127
- # lora_model_dir:
128
- # lora_r: 64
129
- # lora_alpha: 128
130
- # lora_dropout: 0.05
131
- # lora_target_linear: true
132
- # peft_use_dora: true
133
-
134
- unfrozen_parameters:
135
- - ^lm_head.weight$
136
- - ^model.embed_tokens.weight$
137
- # mlp.down_proj layers
138
- - model.layers.62.mlp.down_proj
139
- - model.layers.64.mlp.down_proj
140
- - model.layers.63.mlp.down_proj
141
- - model.layers.66.mlp.down_proj
142
- - model.layers.65.mlp.down_proj
143
- - model.layers.67.mlp.down_proj
144
- - model.layers.68.mlp.down_proj
145
- - model.layers.31.mlp.down_proj
146
- - model.layers.60.mlp.down_proj
147
- - model.layers.69.mlp.down_proj
148
- - model.layers.61.mlp.down_proj
149
- - model.layers.59.mlp.down_proj
150
- - model.layers.30.mlp.down_proj
151
- - model.layers.70.mlp.down_proj
152
- - model.layers.32.mlp.down_proj
153
- - model.layers.34.mlp.down_proj
154
- - model.layers.33.mlp.down_proj
155
- - model.layers.76.mlp.down_proj
156
- - model.layers.72.mlp.down_proj
157
- - model.layers.71.mlp.down_proj
158
- - model.layers.58.mlp.down_proj
159
- - model.layers.75.mlp.down_proj
160
- - model.layers.29.mlp.down_proj
161
- - model.layers.56.mlp.down_proj
162
- - model.layers.26.mlp.down_proj
163
- - model.layers.35.mlp.down_proj
164
- - model.layers.28.mlp.down_proj
165
- - model.layers.57.mlp.down_proj
166
- - model.layers.77.mlp.down_proj
167
- - model.layers.36.mlp.down_proj
168
- - model.layers.27.mlp.down_proj
169
- - model.layers.25.mlp.down_proj
170
- - model.layers.78.mlp.down_proj
171
- - model.layers.37.mlp.down_proj
172
- - model.layers.73.mlp.down_proj
173
- - model.layers.55.mlp.down_proj
174
- - model.layers.54.mlp.down_proj
175
- - model.layers.74.mlp.down_proj
176
- - model.layers.24.mlp.down_proj
177
- - model.layers.53.mlp.down_proj
178
- # mlp.gate_proj layers
179
- - model.layers.78.mlp.gate_proj
180
- - model.layers.77.mlp.gate_proj
181
- - model.layers.76.mlp.gate_proj
182
- - model.layers.79.mlp.gate_proj
183
- - model.layers.75.mlp.gate_proj
184
- - model.layers.74.mlp.gate_proj
185
- - model.layers.73.mlp.gate_proj
186
- - model.layers.72.mlp.gate_proj
187
- - model.layers.71.mlp.gate_proj
188
- - model.layers.70.mlp.gate_proj
189
- - model.layers.69.mlp.gate_proj
190
- - model.layers.57.mlp.gate_proj
191
- - model.layers.54.mlp.gate_proj
192
- - model.layers.55.mlp.gate_proj
193
- - model.layers.68.mlp.gate_proj
194
- - model.layers.63.mlp.gate_proj
195
- - model.layers.53.mlp.gate_proj
196
- - model.layers.44.mlp.gate_proj
197
- - model.layers.45.mlp.gate_proj
198
- - model.layers.49.mlp.gate_proj
199
- - model.layers.58.mlp.gate_proj
200
- - model.layers.46.mlp.gate_proj
201
- - model.layers.56.mlp.gate_proj
202
- - model.layers.67.mlp.gate_proj
203
- - model.layers.62.mlp.gate_proj
204
- - model.layers.50.mlp.gate_proj
205
- - model.layers.64.mlp.gate_proj
206
- - model.layers.52.mlp.gate_proj
207
- - model.layers.40.mlp.gate_proj
208
- - model.layers.43.mlp.gate_proj
209
- - model.layers.48.mlp.gate_proj
210
- - model.layers.66.mlp.gate_proj
211
- - model.layers.47.mlp.gate_proj
212
- - model.layers.59.mlp.gate_proj
213
- - model.layers.65.mlp.gate_proj
214
- - model.layers.61.mlp.gate_proj
215
- - model.layers.60.mlp.gate_proj
216
- - model.layers.42.mlp.gate_proj
217
- - model.layers.51.mlp.gate_proj
218
- - model.layers.41.mlp.gate_proj
219
- # mlp.up_proj layers
220
- - model.layers.70.mlp.up_proj
221
- - model.layers.69.mlp.up_proj
222
- - model.layers.71.mlp.up_proj
223
- - model.layers.68.mlp.up_proj
224
- - model.layers.72.mlp.up_proj
225
- - model.layers.67.mlp.up_proj
226
- - model.layers.66.mlp.up_proj
227
- - model.layers.73.mlp.up_proj
228
- - model.layers.46.mlp.up_proj
229
- - model.layers.63.mlp.up_proj
230
- - model.layers.75.mlp.up_proj
231
- - model.layers.76.mlp.up_proj
232
- - model.layers.74.mlp.up_proj
233
- - model.layers.45.mlp.up_proj
234
- - model.layers.62.mlp.up_proj
235
- - model.layers.64.mlp.up_proj
236
- - model.layers.65.mlp.up_proj
237
- - model.layers.44.mlp.up_proj
238
- - model.layers.53.mlp.up_proj
239
- - model.layers.47.mlp.up_proj
240
- - model.layers.49.mlp.up_proj
241
- - model.layers.48.mlp.up_proj
242
- - model.layers.57.mlp.up_proj
243
- - model.layers.43.mlp.up_proj
244
- - model.layers.42.mlp.up_proj
245
- - model.layers.56.mlp.up_proj
246
- - model.layers.61.mlp.up_proj
247
- - model.layers.54.mlp.up_proj
248
- - model.layers.40.mlp.up_proj
249
- - model.layers.55.mlp.up_proj
250
- - model.layers.77.mlp.up_proj
251
- - model.layers.60.mlp.up_proj
252
- - model.layers.41.mlp.up_proj
253
- - model.layers.35.mlp.up_proj
254
- - model.layers.37.mlp.up_proj
255
- - model.layers.58.mlp.up_proj
256
- - model.layers.34.mlp.up_proj
257
- - model.layers.38.mlp.up_proj
258
- - model.layers.33.mlp.up_proj
259
- - model.layers.39.mlp.up_proj
260
- # self_attn.k_proj layers
261
- - model.layers.36.self_attn.k_proj
262
- - model.layers.79.self_attn.k_proj
263
- - model.layers.35.self_attn.k_proj
264
- - model.layers.34.self_attn.k_proj
265
- - model.layers.37.self_attn.k_proj
266
- - model.layers.33.self_attn.k_proj
267
- - model.layers.38.self_attn.k_proj
268
- - model.layers.39.self_attn.k_proj
269
- - model.layers.74.self_attn.k_proj
270
- - model.layers.77.self_attn.k_proj
271
- - model.layers.41.self_attn.k_proj
272
- - model.layers.69.self_attn.k_proj
273
- - model.layers.32.self_attn.k_proj
274
- - model.layers.78.self_attn.k_proj
275
- - model.layers.30.self_attn.k_proj
276
- - model.layers.70.self_attn.k_proj
277
- - model.layers.25.self_attn.k_proj
278
- - model.layers.42.self_attn.k_proj
279
- - model.layers.29.self_attn.k_proj
280
- - model.layers.31.self_attn.k_proj
281
- - model.layers.68.self_attn.k_proj
282
- - model.layers.66.self_attn.k_proj
283
- - model.layers.22.self_attn.k_proj
284
- - model.layers.65.self_attn.k_proj
285
- - model.layers.44.self_attn.k_proj
286
- - model.layers.40.self_attn.k_proj
287
- - model.layers.63.self_attn.k_proj
288
- - model.layers.23.self_attn.k_proj
289
- - model.layers.28.self_attn.k_proj
290
- - model.layers.24.self_attn.k_proj
291
- - model.layers.26.self_attn.k_proj
292
- - model.layers.67.self_attn.k_proj
293
- - model.layers.75.self_attn.k_proj
294
- - model.layers.27.self_attn.k_proj
295
- - model.layers.57.self_attn.k_proj
296
- - model.layers.64.self_attn.k_proj
297
- - model.layers.71.self_attn.k_proj
298
- - model.layers.61.self_attn.k_proj
299
- - model.layers.72.self_attn.k_proj
300
- - model.layers.73.self_attn.k_proj
301
- # self_attn.o_proj layers
302
- - model.layers.69.self_attn.o_proj
303
- - model.layers.39.self_attn.o_proj
304
- - model.layers.16.self_attn.o_proj
305
- - model.layers.14.self_attn.o_proj
306
- - model.layers.19.self_attn.o_proj
307
- - model.layers.42.self_attn.o_proj
308
- - model.layers.12.self_attn.o_proj
309
- - model.layers.15.self_attn.o_proj
310
- - model.layers.17.self_attn.o_proj
311
- - model.layers.38.self_attn.o_proj
312
- - model.layers.23.self_attn.o_proj
313
- - model.layers.22.self_attn.o_proj
314
- - model.layers.13.self_attn.o_proj
315
- - model.layers.29.self_attn.o_proj
316
- - model.layers.41.self_attn.o_proj
317
- - model.layers.44.self_attn.o_proj
318
- - model.layers.46.self_attn.o_proj
319
- - model.layers.45.self_attn.o_proj
320
- - model.layers.43.self_attn.o_proj
321
- - model.layers.49.self_attn.o_proj
322
- - model.layers.30.self_attn.o_proj
323
- - model.layers.26.self_attn.o_proj
324
- - model.layers.25.self_attn.o_proj
325
- - model.layers.37.self_attn.o_proj
326
- - model.layers.47.self_attn.o_proj
327
- - model.layers.11.self_attn.o_proj
328
- - model.layers.18.self_attn.o_proj
329
- - model.layers.28.self_attn.o_proj
330
- - model.layers.20.self_attn.o_proj
331
- - model.layers.27.self_attn.o_proj
332
- - model.layers.53.self_attn.o_proj
333
- - model.layers.52.self_attn.o_proj
334
- - model.layers.35.self_attn.o_proj
335
- - model.layers.71.self_attn.o_proj
336
- - model.layers.10.self_attn.o_proj
337
- - model.layers.3.self_attn.o_proj
338
- - model.layers.21.self_attn.o_proj
339
- - model.layers.24.self_attn.o_proj
340
- - model.layers.68.self_attn.o_proj
341
- - model.layers.48.self_attn.o_proj
342
- # self_attn.q_proj layers
343
- - model.layers.1.self_attn.q_proj
344
- - model.layers.2.self_attn.q_proj
345
- - model.layers.3.self_attn.q_proj
346
- - model.layers.0.self_attn.q_proj
347
- - model.layers.5.self_attn.q_proj
348
- - model.layers.4.self_attn.q_proj
349
- - model.layers.6.self_attn.q_proj
350
- - model.layers.8.self_attn.q_proj
351
- - model.layers.7.self_attn.q_proj
352
- - model.layers.9.self_attn.q_proj
353
- - model.layers.10.self_attn.q_proj
354
- - model.layers.68.self_attn.q_proj
355
- - model.layers.25.self_attn.q_proj
356
- - model.layers.12.self_attn.q_proj
357
- - model.layers.54.self_attn.q_proj
358
- - model.layers.55.self_attn.q_proj
359
- - model.layers.61.self_attn.q_proj
360
- - model.layers.18.self_attn.q_proj
361
- - model.layers.49.self_attn.q_proj
362
- - model.layers.66.self_attn.q_proj
363
- - model.layers.72.self_attn.q_proj
364
- - model.layers.11.self_attn.q_proj
365
- - model.layers.52.self_attn.q_proj
366
- - model.layers.64.self_attn.q_proj
367
- - model.layers.15.self_attn.q_proj
368
- - model.layers.60.self_attn.q_proj
369
- - model.layers.50.self_attn.q_proj
370
- - model.layers.59.self_attn.q_proj
371
- - model.layers.53.self_attn.q_proj
372
- - model.layers.48.self_attn.q_proj
373
- - model.layers.57.self_attn.q_proj
374
- - model.layers.70.self_attn.q_proj
375
- - model.layers.17.self_attn.q_proj
376
- - model.layers.67.self_attn.q_proj
377
- - model.layers.71.self_attn.q_proj
378
- - model.layers.62.self_attn.q_proj
379
- - model.layers.51.self_attn.q_proj
380
- - model.layers.19.self_attn.q_proj
381
- - model.layers.58.self_attn.q_proj
382
- - model.layers.13.self_attn.q_proj
383
- # self_attn.v_proj layers
384
- - model.layers.23.self_attn.v_proj
385
- - model.layers.25.self_attn.v_proj
386
- - model.layers.26.self_attn.v_proj
387
- - model.layers.27.self_attn.v_proj
388
- - model.layers.28.self_attn.v_proj
389
- - model.layers.29.self_attn.v_proj
390
- - model.layers.30.self_attn.v_proj
391
- - model.layers.31.self_attn.v_proj
392
- - model.layers.34.self_attn.v_proj
393
- - model.layers.35.self_attn.v_proj
394
- - model.layers.36.self_attn.v_proj
395
- - model.layers.37.self_attn.v_proj
396
- - model.layers.38.self_attn.v_proj
397
- - model.layers.42.self_attn.v_proj
398
- - model.layers.48.self_attn.v_proj
399
- - model.layers.57.self_attn.v_proj
400
- - model.layers.58.self_attn.v_proj
401
- - model.layers.61.self_attn.v_proj
402
- - model.layers.63.self_attn.v_proj
403
- - model.layers.64.self_attn.v_proj
404
- - model.layers.65.self_attn.v_proj
405
- - model.layers.66.self_attn.v_proj
406
- - model.layers.69.self_attn.v_proj
407
- - model.layers.70.self_attn.v_proj
408
- - model.layers.74.self_attn.v_proj
409
- - model.layers.75.self_attn.v_proj
410
- - model.layers.72.self_attn.v_proj
411
- - model.layers.39.self_attn.v_proj
412
- - model.layers.41.self_attn.v_proj
413
- - model.layers.40.self_attn.v_proj
414
- - model.layers.33.self_attn.v_proj
415
- - model.layers.59.self_attn.v_proj
416
- - model.layers.16.self_attn.v_proj
417
- - model.layers.15.self_attn.v_proj
418
- - model.layers.76.self_attn.v_proj
419
- - model.layers.24.self_attn.v_proj
420
- - model.layers.68.self_attn.v_proj
421
- - model.layers.67.self_attn.v_proj
422
- - model.layers.55.self_attn.v_proj
423
- - model.layers.44.self_attn.v_proj
424
-
425
-
426
- wandb_project: EVA-Qwen2.5-72B-SFFT-v0.0
427
- wandb_entity:
428
- wandb_watch:
429
- wandb_name: Unit-00
430
- wandb_log_model:
431
-
432
- gradient_accumulation_steps: 4
433
- micro_batch_size: 4
434
- num_epochs: 3
435
- optimizer: paged_adamw_8bit
436
- lr_scheduler: cosine
437
- learning_rate: 0.00005
438
- max_grad_norm: 3
439
-
440
- train_on_inputs: false
441
- group_by_length: false
442
- bf16: auto
443
- fp16:
444
- tf32: false
445
-
446
- gradient_checkpointing: "unsloth"
447
- # gradient_checkpointing_kwargs:
448
- # use_reentrant: true
449
- early_stopping_patience:
450
- resume_from_checkpoint:
451
- local_rank:
452
- logging_steps: 1
453
- xformers_attention:
454
- flash_attention: true
455
-
456
- warmup_steps: 20
457
- evals_per_epoch: 4
458
- saves_per_epoch: 2
459
- save_total_limit: 1
460
- save_safetensors: true
461
- hub_model_id:
462
- hub_strategy:
463
- debug:
464
- deepspeed: deepspeed_configs/zero3_bf16.json
465
- weight_decay: 0.1
466
- # fsdp:
467
- # - full_shard
468
- # - auto_wrap
469
- # fsdp_config:
470
- # fsdp_limit_all_gathers: true
471
- # fsdp_sync_module_states: false
472
- # fsdp_offload_params: true
473
- # fsdp_cpu_ram_efficient_loading: true
474
- # fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
475
- # fsdp_transformer_layer_cls_to_wrap: Qwen2DecoderLayer
476
- # fsdp_activation_checkpointing: true
477
- # fsdp_state_dict_type: SHARDED_STATE_DICT # Changed from FULL_STATE_DICT
478
- # fsdp_sharding_strategy: FULL_SHARD
479
- # fsdp_forward_prefetch: false # Added
480
- # fsdp_backward_prefetch: "BACKWARD_PRE" # Added
481
- # fsdp_backward_prefetch_limit: 1 # Added
482
- # fsdp_mixed_precision: BF16 # Added
483
- ```
484
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
485
  </details><br>
 
1
+ ---
2
+ library_name: transformers
3
+ license: other
4
+ license_name: qwen
5
+ license_link: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE
6
+ base_model: Qwen/Qwen2.5-72B
7
+ datasets:
8
+ - anthracite-org/kalo-opus-instruct-22k-no-refusal
9
+ - Nopm/Opus_WritingStruct
10
+ - Gryphe/Sonnet3.5-SlimOrcaDedupCleaned
11
+ - Gryphe/Sonnet3.5-Charcard-Roleplay
12
+ - Gryphe/ChatGPT-4o-Writing-Prompts
13
+ - Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned
14
+ - Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned
15
+ - nothingiisreal/Reddit-Dirty-And-WritingPrompts
16
+ - allura-org/Celeste-1.x-data-mixture
17
+ tags:
18
+ - generated_from_trainer
19
+ language:
20
+ - zho
21
+ - eng
22
+ - fra
23
+ - spa
24
+ - por
25
+ - deu
26
+ - ita
27
+ - rus
28
+ - jpn
29
+ - kor
30
+ - vie
31
+ - tha
32
+ - ara
33
+ model-index:
34
+ - name: EVA-Qwen2.5-72B-SFFT-v0.0
35
+ results: []
36
+ ---
37
+
38
+ # EVA Qwen2.5-72B v0.0
39
+
40
+ <p>
41
+ A RP/storywriting specialist model, full-parameter finetune of Qwen2.5-72B on mixture of synthetic and natural data.<br>
42
+ It uses Celeste 70B 0.1 data mixture, greatly expanding it to improve versatility, creativity and "flavor" of the resulting model.<br>
43
+ </p>
44
+
45
+ <p>Model is available for inference on <a href=https://featherless.ai/models/EVA-UNIT-01/EVA-Qwen2.5-72B-v0.0>Featherless.AI</a></p
46
+
47
+ <p>Note: using quantized KV cache with Qwen2.5 <b>is not recommended</b> and can lead to degraded output quality. On the other hand, Qwen's KV cache is already light enough, so using f16 for it shouldn't be problematic.</p>
48
+ <p>Note #2: due to some unexpected effects of data normalization, some artifacting in form of randomly appearring sequence of <code>—</code> can appear in outputs sometimes, if penalties are too high. To avoid it, ban token number <code>158</code>. Thanks to Cahvay/ALK for discovering this fix!</p>
49
+ <p>
50
+ <p>Prompt format is ChatML.</p><br>
51
+ <h3>Recommended sampler values:</h3>
52
+ <ul>
53
+ <li>Temperature: 1</li>
54
+ <li>Typical-P: 0.9</li>
55
+ <li>Min-P: 0.05</li>
56
+ <li>Top-A: 0.2</li>
57
+ <li>Repetition Penalty: 1.03</li>
58
+ </ul>
59
+
60
+ <h3>Recommended SillyTavern presets (via CalamitousFelicitousness):</h3>
61
+
62
+ - [Context](https://huggingface.co/EVA-UNIT-01/EVA-Yi-1.5-9B-32K-V1/blob/main/%5BChatML%5D%20Roleplay-v1.9%20Context.json)
63
+ - [Instruct and System Prompt](https://huggingface.co/EVA-UNIT-01/EVA-Yi-1.5-9B-32K-V1/blob/main/%5BChatML%5D%20Roleplay-v1.9%20Instruct.json)
64
+ </p>
65
+
66
+
67
+ <p>
68
+ <br>
69
+ <h3>
70
+ Training data:
71
+ </h3>
72
+ <ul>
73
+ <li>Celeste 70B 0.1 data mixture minus Opus Instruct subset. See that model's <a href=https://huggingface.co/nothingiisreal/L3.1-70B-Celeste-V0.1-BF16>card</a> for details.</li>
74
+ <li>Kalomaze's Opus_Instruct_25k dataset, filtered for refusals.</li>
75
+ <li>A subset (1k rows) of ChatGPT-4o-WritingPrompts by Gryphe</li>
76
+ <li>A subset (2k rows) of Sonnet3.5-Charcards-Roleplay by Gryphe</li>
77
+ <li>Synthstruct and SynthRP datasets by Epiculous</li>
78
+ </ul>
79
+ <h3>
80
+ Training time and hardware:
81
+ </h3>
82
+ <ul><li>12 hours on 8xMI300X</li></ul><br>
83
+ </p>
84
+ <p>Model was trained by Kearm and Auri.</p>
85
+ <h4>Special thanks:</h4><ul>
86
+ <li>to Gryphe, Lemmy, Kalomaze, Nopm and Epiculous for the data</li>
87
+ <li>to CalamitiousFelicitousness for providing free inference for public beta testing</li>
88
+ <li>and to Allura-org for support and feedback on EVA models.</li></ul>
89
+ <a href=https://github.com/axolotl-ai-cloud/axolotl><img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/></a>
90
+ <details><summary>See axolotl config</summary>
91
+
92
+ axolotl version: `0.4.1`
93
+ ```yaml
94
+ base_model: Qwen/Qwen2.5-72B
95
+
96
+ load_in_8bit: false
97
+ load_in_4bit: false
98
+ strict: false
99
+
100
+ plugins:
101
+ - axolotl.integrations.liger.LigerPlugin
102
+ liger_rope: true
103
+ liger_rms_norm: true
104
+ liger_swiglu: false
105
+ liger_fused_linear_cross_entropy: false
106
+
107
+ # plugins:
108
+ # - axolotl.integrations.spectrum.SpectrumPlugin
109
+
110
+ # spectrum_top_fraction: 0.5
111
+ # # Optional if using a pre-scanned model as your base_model. Useful if using a model mirror
112
+ # spectrum_model_name: Qwen/Qwen2.5-32B
113
+
114
+ datasets:
115
+ - path: datasets/deduped_Synthstruct-Gens_processed_sharegpt_converted_cleaned.jsonl
116
+ type: sharegpt
117
+ - path: datasets/opus-instruct-22k-no_refusals-filtered.jsonl
118
+ type: sharegpt
119
+ - path: datasets/Celeste_Filtered.jsonl
120
+ type: sharegpt
121
+ - path: datasets/Gryphe-S3-5-Charcards-names-2k.jsonl
122
+ type: sharegpt
123
+ - path: datasets/deduped_SynthRP-Gens_processed_09-25-2024-ShareGPT_converted_cleaned.jsonl
124
+ type: sharegpt
125
+ - path: datasets/deduped_Gryphe-4o-WP-1k.jsonl
126
+ type: sharegpt
127
+ - path: datasets/deduped_not_samantha_norefusals.jsonl
128
+ type: sharegpt
129
+
130
+ chat_template: chatml
131
+ shuffle_merged_datasets: true
132
+ val_set_size: 0.001
133
+ output_dir: ./EVA-Qwen2.5-72B-SFFT-v0.0
134
+
135
+ sequence_len: 8192
136
+ sample_packing: true
137
+ eval_sample_packing: false
138
+ pad_to_sequence_len: true
139
+
140
+ # adapter: qlora
141
+ # lora_model_dir:
142
+ # lora_r: 64
143
+ # lora_alpha: 128
144
+ # lora_dropout: 0.05
145
+ # lora_target_linear: true
146
+ # peft_use_dora: true
147
+
148
+ unfrozen_parameters:
149
+ - ^lm_head.weight$
150
+ - ^model.embed_tokens.weight$
151
+ # mlp.down_proj layers
152
+ - model.layers.62.mlp.down_proj
153
+ - model.layers.64.mlp.down_proj
154
+ - model.layers.63.mlp.down_proj
155
+ - model.layers.66.mlp.down_proj
156
+ - model.layers.65.mlp.down_proj
157
+ - model.layers.67.mlp.down_proj
158
+ - model.layers.68.mlp.down_proj
159
+ - model.layers.31.mlp.down_proj
160
+ - model.layers.60.mlp.down_proj
161
+ - model.layers.69.mlp.down_proj
162
+ - model.layers.61.mlp.down_proj
163
+ - model.layers.59.mlp.down_proj
164
+ - model.layers.30.mlp.down_proj
165
+ - model.layers.70.mlp.down_proj
166
+ - model.layers.32.mlp.down_proj
167
+ - model.layers.34.mlp.down_proj
168
+ - model.layers.33.mlp.down_proj
169
+ - model.layers.76.mlp.down_proj
170
+ - model.layers.72.mlp.down_proj
171
+ - model.layers.71.mlp.down_proj
172
+ - model.layers.58.mlp.down_proj
173
+ - model.layers.75.mlp.down_proj
174
+ - model.layers.29.mlp.down_proj
175
+ - model.layers.56.mlp.down_proj
176
+ - model.layers.26.mlp.down_proj
177
+ - model.layers.35.mlp.down_proj
178
+ - model.layers.28.mlp.down_proj
179
+ - model.layers.57.mlp.down_proj
180
+ - model.layers.77.mlp.down_proj
181
+ - model.layers.36.mlp.down_proj
182
+ - model.layers.27.mlp.down_proj
183
+ - model.layers.25.mlp.down_proj
184
+ - model.layers.78.mlp.down_proj
185
+ - model.layers.37.mlp.down_proj
186
+ - model.layers.73.mlp.down_proj
187
+ - model.layers.55.mlp.down_proj
188
+ - model.layers.54.mlp.down_proj
189
+ - model.layers.74.mlp.down_proj
190
+ - model.layers.24.mlp.down_proj
191
+ - model.layers.53.mlp.down_proj
192
+ # mlp.gate_proj layers
193
+ - model.layers.78.mlp.gate_proj
194
+ - model.layers.77.mlp.gate_proj
195
+ - model.layers.76.mlp.gate_proj
196
+ - model.layers.79.mlp.gate_proj
197
+ - model.layers.75.mlp.gate_proj
198
+ - model.layers.74.mlp.gate_proj
199
+ - model.layers.73.mlp.gate_proj
200
+ - model.layers.72.mlp.gate_proj
201
+ - model.layers.71.mlp.gate_proj
202
+ - model.layers.70.mlp.gate_proj
203
+ - model.layers.69.mlp.gate_proj
204
+ - model.layers.57.mlp.gate_proj
205
+ - model.layers.54.mlp.gate_proj
206
+ - model.layers.55.mlp.gate_proj
207
+ - model.layers.68.mlp.gate_proj
208
+ - model.layers.63.mlp.gate_proj
209
+ - model.layers.53.mlp.gate_proj
210
+ - model.layers.44.mlp.gate_proj
211
+ - model.layers.45.mlp.gate_proj
212
+ - model.layers.49.mlp.gate_proj
213
+ - model.layers.58.mlp.gate_proj
214
+ - model.layers.46.mlp.gate_proj
215
+ - model.layers.56.mlp.gate_proj
216
+ - model.layers.67.mlp.gate_proj
217
+ - model.layers.62.mlp.gate_proj
218
+ - model.layers.50.mlp.gate_proj
219
+ - model.layers.64.mlp.gate_proj
220
+ - model.layers.52.mlp.gate_proj
221
+ - model.layers.40.mlp.gate_proj
222
+ - model.layers.43.mlp.gate_proj
223
+ - model.layers.48.mlp.gate_proj
224
+ - model.layers.66.mlp.gate_proj
225
+ - model.layers.47.mlp.gate_proj
226
+ - model.layers.59.mlp.gate_proj
227
+ - model.layers.65.mlp.gate_proj
228
+ - model.layers.61.mlp.gate_proj
229
+ - model.layers.60.mlp.gate_proj
230
+ - model.layers.42.mlp.gate_proj
231
+ - model.layers.51.mlp.gate_proj
232
+ - model.layers.41.mlp.gate_proj
233
+ # mlp.up_proj layers
234
+ - model.layers.70.mlp.up_proj
235
+ - model.layers.69.mlp.up_proj
236
+ - model.layers.71.mlp.up_proj
237
+ - model.layers.68.mlp.up_proj
238
+ - model.layers.72.mlp.up_proj
239
+ - model.layers.67.mlp.up_proj
240
+ - model.layers.66.mlp.up_proj
241
+ - model.layers.73.mlp.up_proj
242
+ - model.layers.46.mlp.up_proj
243
+ - model.layers.63.mlp.up_proj
244
+ - model.layers.75.mlp.up_proj
245
+ - model.layers.76.mlp.up_proj
246
+ - model.layers.74.mlp.up_proj
247
+ - model.layers.45.mlp.up_proj
248
+ - model.layers.62.mlp.up_proj
249
+ - model.layers.64.mlp.up_proj
250
+ - model.layers.65.mlp.up_proj
251
+ - model.layers.44.mlp.up_proj
252
+ - model.layers.53.mlp.up_proj
253
+ - model.layers.47.mlp.up_proj
254
+ - model.layers.49.mlp.up_proj
255
+ - model.layers.48.mlp.up_proj
256
+ - model.layers.57.mlp.up_proj
257
+ - model.layers.43.mlp.up_proj
258
+ - model.layers.42.mlp.up_proj
259
+ - model.layers.56.mlp.up_proj
260
+ - model.layers.61.mlp.up_proj
261
+ - model.layers.54.mlp.up_proj
262
+ - model.layers.40.mlp.up_proj
263
+ - model.layers.55.mlp.up_proj
264
+ - model.layers.77.mlp.up_proj
265
+ - model.layers.60.mlp.up_proj
266
+ - model.layers.41.mlp.up_proj
267
+ - model.layers.35.mlp.up_proj
268
+ - model.layers.37.mlp.up_proj
269
+ - model.layers.58.mlp.up_proj
270
+ - model.layers.34.mlp.up_proj
271
+ - model.layers.38.mlp.up_proj
272
+ - model.layers.33.mlp.up_proj
273
+ - model.layers.39.mlp.up_proj
274
+ # self_attn.k_proj layers
275
+ - model.layers.36.self_attn.k_proj
276
+ - model.layers.79.self_attn.k_proj
277
+ - model.layers.35.self_attn.k_proj
278
+ - model.layers.34.self_attn.k_proj
279
+ - model.layers.37.self_attn.k_proj
280
+ - model.layers.33.self_attn.k_proj
281
+ - model.layers.38.self_attn.k_proj
282
+ - model.layers.39.self_attn.k_proj
283
+ - model.layers.74.self_attn.k_proj
284
+ - model.layers.77.self_attn.k_proj
285
+ - model.layers.41.self_attn.k_proj
286
+ - model.layers.69.self_attn.k_proj
287
+ - model.layers.32.self_attn.k_proj
288
+ - model.layers.78.self_attn.k_proj
289
+ - model.layers.30.self_attn.k_proj
290
+ - model.layers.70.self_attn.k_proj
291
+ - model.layers.25.self_attn.k_proj
292
+ - model.layers.42.self_attn.k_proj
293
+ - model.layers.29.self_attn.k_proj
294
+ - model.layers.31.self_attn.k_proj
295
+ - model.layers.68.self_attn.k_proj
296
+ - model.layers.66.self_attn.k_proj
297
+ - model.layers.22.self_attn.k_proj
298
+ - model.layers.65.self_attn.k_proj
299
+ - model.layers.44.self_attn.k_proj
300
+ - model.layers.40.self_attn.k_proj
301
+ - model.layers.63.self_attn.k_proj
302
+ - model.layers.23.self_attn.k_proj
303
+ - model.layers.28.self_attn.k_proj
304
+ - model.layers.24.self_attn.k_proj
305
+ - model.layers.26.self_attn.k_proj
306
+ - model.layers.67.self_attn.k_proj
307
+ - model.layers.75.self_attn.k_proj
308
+ - model.layers.27.self_attn.k_proj
309
+ - model.layers.57.self_attn.k_proj
310
+ - model.layers.64.self_attn.k_proj
311
+ - model.layers.71.self_attn.k_proj
312
+ - model.layers.61.self_attn.k_proj
313
+ - model.layers.72.self_attn.k_proj
314
+ - model.layers.73.self_attn.k_proj
315
+ # self_attn.o_proj layers
316
+ - model.layers.69.self_attn.o_proj
317
+ - model.layers.39.self_attn.o_proj
318
+ - model.layers.16.self_attn.o_proj
319
+ - model.layers.14.self_attn.o_proj
320
+ - model.layers.19.self_attn.o_proj
321
+ - model.layers.42.self_attn.o_proj
322
+ - model.layers.12.self_attn.o_proj
323
+ - model.layers.15.self_attn.o_proj
324
+ - model.layers.17.self_attn.o_proj
325
+ - model.layers.38.self_attn.o_proj
326
+ - model.layers.23.self_attn.o_proj
327
+ - model.layers.22.self_attn.o_proj
328
+ - model.layers.13.self_attn.o_proj
329
+ - model.layers.29.self_attn.o_proj
330
+ - model.layers.41.self_attn.o_proj
331
+ - model.layers.44.self_attn.o_proj
332
+ - model.layers.46.self_attn.o_proj
333
+ - model.layers.45.self_attn.o_proj
334
+ - model.layers.43.self_attn.o_proj
335
+ - model.layers.49.self_attn.o_proj
336
+ - model.layers.30.self_attn.o_proj
337
+ - model.layers.26.self_attn.o_proj
338
+ - model.layers.25.self_attn.o_proj
339
+ - model.layers.37.self_attn.o_proj
340
+ - model.layers.47.self_attn.o_proj
341
+ - model.layers.11.self_attn.o_proj
342
+ - model.layers.18.self_attn.o_proj
343
+ - model.layers.28.self_attn.o_proj
344
+ - model.layers.20.self_attn.o_proj
345
+ - model.layers.27.self_attn.o_proj
346
+ - model.layers.53.self_attn.o_proj
347
+ - model.layers.52.self_attn.o_proj
348
+ - model.layers.35.self_attn.o_proj
349
+ - model.layers.71.self_attn.o_proj
350
+ - model.layers.10.self_attn.o_proj
351
+ - model.layers.3.self_attn.o_proj
352
+ - model.layers.21.self_attn.o_proj
353
+ - model.layers.24.self_attn.o_proj
354
+ - model.layers.68.self_attn.o_proj
355
+ - model.layers.48.self_attn.o_proj
356
+ # self_attn.q_proj layers
357
+ - model.layers.1.self_attn.q_proj
358
+ - model.layers.2.self_attn.q_proj
359
+ - model.layers.3.self_attn.q_proj
360
+ - model.layers.0.self_attn.q_proj
361
+ - model.layers.5.self_attn.q_proj
362
+ - model.layers.4.self_attn.q_proj
363
+ - model.layers.6.self_attn.q_proj
364
+ - model.layers.8.self_attn.q_proj
365
+ - model.layers.7.self_attn.q_proj
366
+ - model.layers.9.self_attn.q_proj
367
+ - model.layers.10.self_attn.q_proj
368
+ - model.layers.68.self_attn.q_proj
369
+ - model.layers.25.self_attn.q_proj
370
+ - model.layers.12.self_attn.q_proj
371
+ - model.layers.54.self_attn.q_proj
372
+ - model.layers.55.self_attn.q_proj
373
+ - model.layers.61.self_attn.q_proj
374
+ - model.layers.18.self_attn.q_proj
375
+ - model.layers.49.self_attn.q_proj
376
+ - model.layers.66.self_attn.q_proj
377
+ - model.layers.72.self_attn.q_proj
378
+ - model.layers.11.self_attn.q_proj
379
+ - model.layers.52.self_attn.q_proj
380
+ - model.layers.64.self_attn.q_proj
381
+ - model.layers.15.self_attn.q_proj
382
+ - model.layers.60.self_attn.q_proj
383
+ - model.layers.50.self_attn.q_proj
384
+ - model.layers.59.self_attn.q_proj
385
+ - model.layers.53.self_attn.q_proj
386
+ - model.layers.48.self_attn.q_proj
387
+ - model.layers.57.self_attn.q_proj
388
+ - model.layers.70.self_attn.q_proj
389
+ - model.layers.17.self_attn.q_proj
390
+ - model.layers.67.self_attn.q_proj
391
+ - model.layers.71.self_attn.q_proj
392
+ - model.layers.62.self_attn.q_proj
393
+ - model.layers.51.self_attn.q_proj
394
+ - model.layers.19.self_attn.q_proj
395
+ - model.layers.58.self_attn.q_proj
396
+ - model.layers.13.self_attn.q_proj
397
+ # self_attn.v_proj layers
398
+ - model.layers.23.self_attn.v_proj
399
+ - model.layers.25.self_attn.v_proj
400
+ - model.layers.26.self_attn.v_proj
401
+ - model.layers.27.self_attn.v_proj
402
+ - model.layers.28.self_attn.v_proj
403
+ - model.layers.29.self_attn.v_proj
404
+ - model.layers.30.self_attn.v_proj
405
+ - model.layers.31.self_attn.v_proj
406
+ - model.layers.34.self_attn.v_proj
407
+ - model.layers.35.self_attn.v_proj
408
+ - model.layers.36.self_attn.v_proj
409
+ - model.layers.37.self_attn.v_proj
410
+ - model.layers.38.self_attn.v_proj
411
+ - model.layers.42.self_attn.v_proj
412
+ - model.layers.48.self_attn.v_proj
413
+ - model.layers.57.self_attn.v_proj
414
+ - model.layers.58.self_attn.v_proj
415
+ - model.layers.61.self_attn.v_proj
416
+ - model.layers.63.self_attn.v_proj
417
+ - model.layers.64.self_attn.v_proj
418
+ - model.layers.65.self_attn.v_proj
419
+ - model.layers.66.self_attn.v_proj
420
+ - model.layers.69.self_attn.v_proj
421
+ - model.layers.70.self_attn.v_proj
422
+ - model.layers.74.self_attn.v_proj
423
+ - model.layers.75.self_attn.v_proj
424
+ - model.layers.72.self_attn.v_proj
425
+ - model.layers.39.self_attn.v_proj
426
+ - model.layers.41.self_attn.v_proj
427
+ - model.layers.40.self_attn.v_proj
428
+ - model.layers.33.self_attn.v_proj
429
+ - model.layers.59.self_attn.v_proj
430
+ - model.layers.16.self_attn.v_proj
431
+ - model.layers.15.self_attn.v_proj
432
+ - model.layers.76.self_attn.v_proj
433
+ - model.layers.24.self_attn.v_proj
434
+ - model.layers.68.self_attn.v_proj
435
+ - model.layers.67.self_attn.v_proj
436
+ - model.layers.55.self_attn.v_proj
437
+ - model.layers.44.self_attn.v_proj
438
+
439
+
440
+ wandb_project: EVA-Qwen2.5-72B-SFFT-v0.0
441
+ wandb_entity:
442
+ wandb_watch:
443
+ wandb_name: Unit-00
444
+ wandb_log_model:
445
+
446
+ gradient_accumulation_steps: 4
447
+ micro_batch_size: 4
448
+ num_epochs: 3
449
+ optimizer: paged_adamw_8bit
450
+ lr_scheduler: cosine
451
+ learning_rate: 0.00005
452
+ max_grad_norm: 3
453
+
454
+ train_on_inputs: false
455
+ group_by_length: false
456
+ bf16: auto
457
+ fp16:
458
+ tf32: false
459
+
460
+ gradient_checkpointing: "unsloth"
461
+ # gradient_checkpointing_kwargs:
462
+ # use_reentrant: true
463
+ early_stopping_patience:
464
+ resume_from_checkpoint:
465
+ local_rank:
466
+ logging_steps: 1
467
+ xformers_attention:
468
+ flash_attention: true
469
+
470
+ warmup_steps: 20
471
+ evals_per_epoch: 4
472
+ saves_per_epoch: 2
473
+ save_total_limit: 1
474
+ save_safetensors: true
475
+ hub_model_id:
476
+ hub_strategy:
477
+ debug:
478
+ deepspeed: deepspeed_configs/zero3_bf16.json
479
+ weight_decay: 0.1
480
+ # fsdp:
481
+ # - full_shard
482
+ # - auto_wrap
483
+ # fsdp_config:
484
+ # fsdp_limit_all_gathers: true
485
+ # fsdp_sync_module_states: false
486
+ # fsdp_offload_params: true
487
+ # fsdp_cpu_ram_efficient_loading: true
488
+ # fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
489
+ # fsdp_transformer_layer_cls_to_wrap: Qwen2DecoderLayer
490
+ # fsdp_activation_checkpointing: true
491
+ # fsdp_state_dict_type: SHARDED_STATE_DICT # Changed from FULL_STATE_DICT
492
+ # fsdp_sharding_strategy: FULL_SHARD
493
+ # fsdp_forward_prefetch: false # Added
494
+ # fsdp_backward_prefetch: "BACKWARD_PRE" # Added
495
+ # fsdp_backward_prefetch_limit: 1 # Added
496
+ # fsdp_mixed_precision: BF16 # Added
497
+ ```
498
+
499
  </details><br>