Siqi-Hu/Llama2-7B-lora-r-32-generic-step-1050-lr-1e-5-labels_40.0-optimized

Llama2-7B-lora-r-32-generic-step-1200-lr-1e-5-labels_40.0-3

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.8089

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 1200
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
5.4776	0.9132	50	4.8849
4.2584	1.8265	100	3.8766
3.674	2.7397	150	3.5260
3.3895	3.6530	200	3.3094
3.2045	4.5662	250	3.1682
3.0743	5.4795	300	3.0658
2.944	6.3927	350	2.9913
2.8487	7.3059	400	2.9373
2.7773	8.2192	450	2.8989
2.7123	9.1324	500	2.8699
2.6538	10.0457	550	2.8497
2.6145	10.9589	600	2.8358
2.5768	11.8721	650	2.8264
2.5979	12.7854	700	2.8210
2.5595	13.6986	750	2.8141
2.5285	14.6119	800	2.8104
2.5112	15.5251	850	2.8109
2.4877	16.4384	900	2.8095
2.4536	17.3516	950	2.8070
2.5073	18.2648	1000	2.8081
2.4321	19.1781	1050	2.8069
2.4284	20.0913	1100	2.8085
2.4627	21.0046	1150	2.8088
2.4636	21.9178	1200	2.8089

Framework versions

• PEFT 0.15.2
• Transformers 4.45.2
• Pytorch 2.5.0+cu121
• Datasets 3.2.0
• Tokenizers 0.20.3