[tokenizer] AlternateTokenizer - a minpeter Collection

minpeter 's Collections

[Dataset] K-Corpus

[Dataset] FineWeb2 Edu Korean

[Model] Very, very small things

[Dataset] Pretrain-corpus

[Model] en-ko trans

[Dataset] Candidate datasets to translate

[Dataset] common-pile korean (Filtered-raw)

[Study] NN MNIST

[Model] FLUX.1 Full Finetuned & Merged

[🛠️] Huggingface Utility

[Dataset] unified standard function calling

[tokenizer] AlternateTokenizer

[Dataset] Function Calling

[tokenizer] AlternateTokenizer

updated Mar 15

A series of modified tokenizers for tuning chatml inspired by teknium and hermes.

minpeter/Llama-3.x-AlternateTokenizer

Updated Feb 11 • 1

Note An alternative tokenizer for the final form of the Llama 3.x series.
teknium/Llama-3.1-AlternateTokenizer

Text Generation • 8B • Updated Jun 18 • 16 • • 4
minpeter/Llama-3.2-1B-AlternateTokenizer-chatml

Text Generation • 1B • Updated Feb 9 • 3
minpeter/Llama-3.2-1B-AlternateTokenizer-tool-chatml

Text Generation • 1B • Updated Feb 10 • 1
philschmid/gemma-tokenizer-chatml

Updated Feb 29, 2024 • 32