README.md

metadata

license: apache-2.0
base_model: binh230/mambaformer
tags:
  - generated_from_trainer
metrics:
  - accuracy
  - precision
  - recall
  - f1
model-index:
  - name: mambaformer
    results: []

mambaformer

This model is a fine-tuned version of binh230/mambaformer on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4950
• Accuracy: 0.7747
• Precision: 0.8314
• Recall: 0.7747
• F1: 0.7647
• Auroc: 0.9429

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 64
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1
• mixed_precision_training: Native AMP
• label_smoothing_factor: 0.03

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1	Auroc
0.6026	0.0471	256	0.5927	0.6776	0.7982	0.6776	0.6414	0.9217
0.3481	0.0941	512	0.4428	0.8167	0.8239	0.8167	0.8157	0.9015
0.2434	0.1412	768	0.4749	0.7833	0.8043	0.7833	0.7795	0.8934
0.1975	0.1882	1024	0.5786	0.7304	0.7949	0.7304	0.7149	0.8979
0.1749	0.2353	1280	0.6214	0.7157	0.7952	0.7157	0.6952	0.9004
0.1644	0.2824	1536	0.6323	0.7107	0.7984	0.7107	0.6877	0.9123
0.1556	0.3294	1792	0.6491	0.7046	0.7990	0.7046	0.6793	0.9161
0.1476	0.3765	2048	0.6989	0.6884	0.7955	0.6884	0.6573	0.9203
0.1466	0.4235	2304	0.6633	0.7014	0.8016	0.7014	0.6744	0.9241
0.1405	0.4706	2560	0.6076	0.7229	0.8071	0.7229	0.7026	0.9250
0.1399	0.5177	2816	0.6221	0.7164	0.8042	0.7164	0.6943	0.9248
0.135	0.5647	3072	0.6249	0.7150	0.8064	0.7150	0.6920	0.9290
0.1339	0.6118	3328	0.6109	0.7233	0.8108	0.7233	0.7024	0.9340
0.1297	0.6589	3584	0.5931	0.7306	0.8127	0.7306	0.7117	0.9360
0.1298	0.7059	3840	0.5644	0.7439	0.8170	0.7439	0.7282	0.9357
0.1275	0.7530	4096	0.5526	0.7475	0.8209	0.7475	0.7322	0.9416
0.1268	0.8000	4352	0.5564	0.7470	0.8203	0.7470	0.7317	0.9412
0.1235	0.8471	4608	0.5439	0.7537	0.8238	0.7537	0.7396	0.9436
0.1231	0.8942	4864	0.5051	0.7693	0.8292	0.7693	0.7583	0.9446
0.1222	0.9412	5120	0.5254	0.7611	0.8241	0.7611	0.7488	0.9405
0.1198	0.9883	5376	0.5439	0.7534	0.8230	0.7534	0.7394	0.9408

Framework versions

• Transformers 4.43.0.dev0
• Pytorch 2.4.0+cu124
• Datasets 2.19.1
• Tokenizers 0.19.1