Chinse_BBPE_Vocab / tokenizer_config.json
Hengzongshu's picture
Upload 2 files
5db4899 verified
{
"special_tokens": {
"bos_token": "<s>",
"pad_token": "<pad>",
"eos_token": "</s>",
"unk_token": "<unk>",
"mask_token": "<mask>"
},
"additional_special_tokens": []
}