Update README.md
Browse files
README.md
CHANGED
|
@@ -12,9 +12,9 @@ widget:
|
|
| 12 |
|
| 13 |
## Model description
|
| 14 |
|
| 15 |
-
This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658).
|
| 16 |
|
| 17 |
-
[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users
|
| 18 |
|
| 19 |
You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:
|
| 20 |
|
|
@@ -189,6 +189,14 @@ python3 scripts/convert_bert_from_uer_to_huggingface.py --input_model_path model
|
|
| 189 |
pages={241},
|
| 190 |
year={2019}
|
| 191 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 192 |
```
|
| 193 |
|
| 194 |
[2_128]:https://huggingface.co/uer/chinese_roberta_L-2_H-128
|
|
|
|
| 12 |
|
| 13 |
## Model description
|
| 14 |
|
| 15 |
+
This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://github.com/dbiir/UER-py/), which is introduced in [this paper](https://arxiv.org/abs/1909.05658). Besides, the models could also be pre-trained by [TencentPretrain](https://github.com/Tencent/TencentPretrain) introduced in [this paper](https://arxiv.org/pdf/2212.06385.pdf), which inherits [UER-py](https://github.com/dbiir/UER-py/) to support models with parameters above one billion, and extends it to a multimodal pre-training framework.
|
| 16 |
|
| 17 |
+
[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users in reproducing the results, we used a publicly available corpus and provided all training details.
|
| 18 |
|
| 19 |
You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the links below:
|
| 20 |
|
|
|
|
| 189 |
pages={241},
|
| 190 |
year={2019}
|
| 191 |
}
|
| 192 |
+
|
| 193 |
+
@article{zhao2023tencentpretrain,
|
| 194 |
+
title={TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities},
|
| 195 |
+
author={Zhao, Zhe and Li, Yudong and Hou, Cheng and Zhao, Jing and others},
|
| 196 |
+
journal={ACL 2023},
|
| 197 |
+
pages={217},
|
| 198 |
+
year={2023}
|
| 199 |
+
}
|
| 200 |
```
|
| 201 |
|
| 202 |
[2_128]:https://huggingface.co/uer/chinese_roberta_L-2_H-128
|