Hugging Face's logo

diagonalge
/
Covanent-Base-Non-Chat

Safetensors
llama
Model card Files
xet
Community
diagonalge commited on
Commit
497725c
·
verified ·
1 Parent(s): 31f9bae

Import from tplr/Covenant72B@Checkpoint-Two into main

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +2 -0
  2. README.md +50 -0
  3. assets/checkpoint-one.webp +3 -0
  4. config.json +29 -0
  5. generation_config.json +7 -0
  6. model-00001-of-00062.safetensors +3 -0
  7. model-00002-of-00062.safetensors +3 -0
  8. model-00003-of-00062.safetensors +3 -0
  9. model-00004-of-00062.safetensors +3 -0
  10. model-00005-of-00062.safetensors +3 -0
  11. model-00006-of-00062.safetensors +3 -0
  12. model-00007-of-00062.safetensors +3 -0
  13. model-00008-of-00062.safetensors +3 -0
  14. model-00009-of-00062.safetensors +3 -0
  15. model-00010-of-00062.safetensors +3 -0
  16. model-00011-of-00062.safetensors +3 -0
  17. model-00012-of-00062.safetensors +3 -0
  18. model-00013-of-00062.safetensors +3 -0
  19. model-00014-of-00062.safetensors +3 -0
  20. model-00015-of-00062.safetensors +3 -0
  21. model-00016-of-00062.safetensors +3 -0
  22. model-00017-of-00062.safetensors +3 -0
  23. model-00018-of-00062.safetensors +3 -0
  24. model-00019-of-00062.safetensors +3 -0
  25. model-00020-of-00062.safetensors +3 -0
  26. model-00021-of-00062.safetensors +3 -0
  27. model-00022-of-00062.safetensors +3 -0
  28. model-00023-of-00062.safetensors +3 -0
  29. model-00024-of-00062.safetensors +3 -0
  30. model-00025-of-00062.safetensors +3 -0
  31. model-00026-of-00062.safetensors +3 -0
  32. model-00027-of-00062.safetensors +3 -0
  33. model-00028-of-00062.safetensors +3 -0
  34. model-00029-of-00062.safetensors +3 -0
  35. model-00030-of-00062.safetensors +3 -0
  36. model-00031-of-00062.safetensors +3 -0
  37. model-00032-of-00062.safetensors +3 -0
  38. model-00033-of-00062.safetensors +3 -0
  39. model-00034-of-00062.safetensors +3 -0
  40. model-00035-of-00062.safetensors +3 -0
  41. model-00036-of-00062.safetensors +3 -0
  42. model-00037-of-00062.safetensors +3 -0
  43. model-00038-of-00062.safetensors +3 -0
  44. model-00039-of-00062.safetensors +3 -0
  45. model-00040-of-00062.safetensors +3 -0
  46. model-00041-of-00062.safetensors +3 -0
  47. model-00042-of-00062.safetensors +3 -0
  48. model-00043-of-00062.safetensors +3 -0
  49. model-00044-of-00062.safetensors +3 -0
  50. model-00045-of-00062.safetensors +3 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ *.webp filter=lfs diff=lfs merge=lfs -text
37
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ datasets:
4
+ - mlfoundations/dclm-baseline-1.0-parquet
5
+ ---
6
+
7
+ # Covenant72B
8
+
9
+ **Covenant72B** is the largest permissionless collaboratively trained language
10
+ model trained entirely from scratch at the 72 billion parameter scale.
11
+
12
+ It is being trained with 20+ globally distributed participants coordinated via
13
+ decentralized infrastructure on the Bittensor blockchain.
14
+
15
+ **Checkpoint-One** marks the first release, corresponding to **200 billion
16
+ tokens processed**. Model files are available in the [Checkpoint-One
17
+ branch](https://huggingface.co/tplr/Covenant72B/tree/Checkpoint-One). Future
18
+ checkpoints will be updated here.
19
+
20
+ ![Checkpoint One](assets/checkpoint-one.webp)
21
+
22
+ ---
23
+
24
+ ## Training Details
25
+
26
+ | Property | Value |
27
+ |-----------|--------|
28
+ | **Model size** | 72B |
29
+ | **Architecture** | LLaMA-style |
30
+ | **Target token budget** | 1.2T (210B for current checkpoint) |
31
+ | **Compute participants** | 20+ |
32
+ | **Minimal compute per participant** | 8×B200 or equivalent |
33
+ | **Dataset** | DCLM-baseline |
34
+ | **Optimizer** | SparseLoCo (communication-efficient optimizer) |
35
+
36
+ ---
37
+
38
+ ## Performance on Benchmarks
39
+ _All results are 0-shot acc-norm (%)_
40
+
41
+ | Model | Compute Environment / Permissions | Size | Tokens | ARC-C | ARC-E | PIQA | OpenBookQA | HellaSwag | Winogrande | MMLU |
42
+ |:------|:----------------------------------|------:|--------:|------:|------:|------:|------------:|-----------:|-------------:|------:|
43
+ | **Intellect-1** | Over the internet / White List | 10B | 1T | 44.8 | 71.6 | 77.7 | 43.6 | 70.5 | 63.1 | 32.7 |
44
+ | **Psyche Consilience-7Y9** | Over the internet / White List | 40B | 1.2T | 31.1 | 55.8 | 76.1 | 34.8 | 63.7 | 57.0 | 24.2 |
45
+ | **Covenant72B – Checkpoint One** | Over the internet / Permissionless | 70B | 210B | 46.2 | 72.6 | 79.2 | 43.0 | 73.5 | 70.3 | 38.0 |
46
+ | **K2 Checkpoint 54** | Centralized Cluster | 65B | 210B | 41.8 | 69.5 | 80.1 | 42.4 | 74.9 | 68.9 | 33.7 |
47
+
48
+ ---
49
+
50
+ For more details, refer to [Checkpoint One on Templar Research](https://templarresearch.substack.com/p/checkpoint-one).
assets/checkpoint-one.webp ADDED

Git LFS Details

  • SHA256: 8dc38984c1e5502a79a1d0409bd309896250de2d50660f81ef375f578c67d9a9
  • Pointer size: 131 Bytes
  • Size of remote file: 585 kB
config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "dtype": "float32",
9
+ "eos_token_id": 2,
10
+ "head_dim": 128,
11
+ "hidden_act": "silu",
12
+ "hidden_size": 8192,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 28672,
15
+ "max_position_embeddings": 2048,
16
+ "mlp_bias": false,
17
+ "model_type": "llama",
18
+ "num_attention_heads": 64,
19
+ "num_hidden_layers": 80,
20
+ "num_key_value_heads": 8,
21
+ "pretraining_tp": 1,
22
+ "rms_norm_eps": 1e-06,
23
+ "rope_scaling": null,
24
+ "rope_theta": 10000.0,
25
+ "tie_word_embeddings": false,
26
+ "transformers_version": "4.56.1",
27
+ "use_cache": false,
28
+ "vocab_size": 262144
29
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.56.1",
6
+ "use_cache": false
7
+ }
model-00001-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:23ca391ac281fbced62b7d0e4e3628b5a2ec81913c790b6afb30a48a305c0076
3
+ size 8589934728
model-00002-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b033e2893d2b786176f75fbdefc8359116d8751ca94b6d0eca89c45611b1d559
3
+ size 4966123104
model-00003-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9307a403433e35af902771d150e5bd406d2218d9c84a2db8882817a0ffccef91
3
+ size 4362142864
model-00004-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b55517d0f875a12171a32a2f06fc694101e82d2d4df6b99cd302c5722e812cc
3
+ size 4966188864
model-00005-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:db233477fe746ad126c9042e995fdfbbd7f87e5320d68cc802c41e6cea87b891
3
+ size 4362142864
model-00006-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:42c78c0868d1997f748eaec3869de2a215c469f69924e1aecc73dcdc0eca82be
3
+ size 4362142864
model-00007-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:97a2bcda0f24803bc097ce32b4db6b23cc2881a184567d3fa2c99f5605f0dd16
3
+ size 4966188864
model-00008-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:285cadd3e04776e033b99ccfb35ef65717e4d62684f031ec6ed582ad7946197c
3
+ size 4362142864
model-00009-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:632480afc3beb01e67479487f7f95cd4f30a9a868fc4009b72bd5cc9f4f29f81
3
+ size 4362142880
model-00010-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8643c59a716f0223df715386526f1c25c05bf957b789687479e48bfabed338cd
3
+ size 4966188880
model-00011-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2c35de08f67f35104af5ec4925b9b519f4976f1c7979944c86195838cd06ff92
3
+ size 4362142872
model-00012-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:598596f09c88744412b6348901fdba2bd44a316ccdfc31157622ca23372b861c
3
+ size 4362142872
model-00013-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:679dd07867aef776d9c592eaa2c8e94814af14f414832b9e5e10ebeb7ddb126c
3
+ size 4966188880
model-00014-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d97763c593fe0f499c02e0f8b5d38c6d391636efb007b835fc3ca84bf0b1c6cc
3
+ size 4362142872
model-00015-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84784c9bba593d672f589ad79e690f2ea60dc2c969bab11cc6d59a4550118b14
3
+ size 4362142872
model-00016-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8d576d63ad15ff00448750019653cd88b0f48967d6367519f9c29610fa8db58a
3
+ size 4966188880
model-00017-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e4ab565556d7e975df936e57e3035acfb7cf424c13e65167cd9bfa1744a111e3
3
+ size 4362142872
model-00018-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15f2c762ea81f6f52781c5ad2bed03636a2b8646f0e665af9bb894c1b1fb1bc0
3
+ size 4362142872
model-00019-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:791e310196bc00555c7ed2be45b42d54b84a796612acb11ffb8059c5eda3962f
3
+ size 4966188880
model-00020-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1cf850bcd900bde059ab2143848fe00d84eba0227ecc9d189523363431a86ca5
3
+ size 4362142872
model-00021-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c2ebe3e716779bee4dfc7f9766d63f7323df025fafd1878775b5af5c7eee8c1
3
+ size 4362142872
model-00022-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6d76a7ace99a67fe2d3379dcd617f7900a36ecf779766f5aeea28c15d434b451
3
+ size 4966188880
model-00023-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c6be12d1269c239575e370425912e956cad788ae95a9dbaf6aa9e711bc8a680b
3
+ size 4362142872
model-00024-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f4d10b6d1bea22a8dbe7053691bb19d7a9086d048b9ef6717db37c93127fff39
3
+ size 4362142872
model-00025-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6620c3d8e9055ea2a95160c483a8d72d1b27f8b38abf23388021e5003e0846fa
3
+ size 4966188880
model-00026-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:069e712adc26add1bb409c9fee4b09cefd02497f1a3fd8a4c5e4786961de9f52
3
+ size 4362142872
model-00027-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:153ca05c610b88e579880bba2f6b18070cb604a7cda4f9fb58b0551f0d1cc3d0
3
+ size 4362142872
model-00028-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:688e38b205f72b9fc2928d4da8690995cca2ab12d19b9c53e10622daa95219a2
3
+ size 4966188880
model-00029-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2c1a1973bf6177ee51c0bcd4815d2595bc475c97906d747b906a425e2d5dfb34
3
+ size 4362142872
model-00030-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a83f0da5ac47d4dc90a4df2ef4ec71df7b8d0fc8616223d0f9532c24b0b02ede
3
+ size 4362142872
model-00031-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:402d1551836254cda46ab5af11a0df51bf14aff36c4a15f9307a9019e3484228
3
+ size 4966188880
model-00032-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e6c1924a8deffd54315c3e234c5f559c5dd1fd0f3629cbe7644f6ba32624aee8
3
+ size 4362142872
model-00033-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:113f0a5b423013a2d6c87e726ec7b7272395b5e8d1e0cea1466def1e00f67d99
3
+ size 4362142872
model-00034-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9680f27f9aa1b3eb35f9807bd2667d54adbc7a40141c3bf333bbefcc2e7c3e05
3
+ size 4966188880
model-00035-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1bef1ca0cfb77d20898403358dbcec4fc7a17de23bf59682c17dc344060a340f
3
+ size 4362142872
model-00036-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:22c39c92c2c70b13ec409c8af462566466a2d368382249281144d27895cfca98
3
+ size 4362142872
model-00037-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:87852971d494373f262f2aa5e01e055e91b0cba804fde8cf7bbb8fe8fe16357f
3
+ size 4966188880
model-00038-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8eb9ad07210472b1ca281f9fd692246a45936611756d44ad128c520a814b849c
3
+ size 4362142872
model-00039-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:733dc1765c2bfce4905b8dd826b7c852f4801651f468c3ad71015e89b484e4bb
3
+ size 4362142872
model-00040-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:846352f8e9ac476d66b3d674eb40f86299d2131388132853d4ef3944c693e9af
3
+ size 4966188880
model-00041-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c990eb5a97c7baf09e67bcb7db6f26bb922bf1f55ff8d9e12796e827311179a
3
+ size 4362142872
model-00042-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:33514dfae257d8b633e36496775c539b506a2e5e27ce88711842d56200d2aaf1
3
+ size 4362142872
model-00043-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2c757f6851cb47b1b035fd627bc72e07f37bae19c45bea96b0f13e97bc6a423a
3
+ size 4966188880
model-00044-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3bc26391007adf3d5e8d93637efeb1cf57fbcc34fc878fb741d7f20ed9863e9b
3
+ size 4362142872
model-00045-of-00062.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:766bb4f3d775a577adb90bd0ca5178674a0c4d9d5862de4c76e72b40d7a93e25
3
+ size 4362142872