Model save

README.md

@@ -0,0 +1,68 @@
+---
+base_model: alignment-handbook/zephyr-7b-sft-full
+library_name: transformers
+model_name: ultrafeedbackSkyworkAgree_alignmentZephyr7BSftFull_sdpo_score_ebs256_lr5e-07_0
+tags:
+- generated_from_trainer
+- trl
+- dpo
+licence: license
+---
+
+# Model Card for ultrafeedbackSkyworkAgree_alignmentZephyr7BSftFull_sdpo_score_ebs256_lr5e-07_0
+
+This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+
+## Quick start
+
+```python
+from transformers import pipeline
+
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="YuchenLi01/ultrafeedbackSkyworkAgree_alignmentZephyr7BSftFull_sdpo_score_ebs256_lr5e-07_0", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+
+## Training procedure
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/yuchenl4/lmpref/runs/ultrafeedbackSkyworkAgree_alignmentZephyr7BSftFull_sdpo_score_ebs256_lr5e-07_0try1G5zGRd9yfHqyQ17rXoMK8LndscAvhddrNANkigW0gCiiC)
+
+This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
+
+### Framework versions
+
+- TRL: 0.12.0
+- Transformers: 4.46.2
+- Pytorch: 2.3.0
+- Datasets: 3.1.0
+- Tokenizers: 0.20.3
+
+## Citations
+
+Cite DPO as:
+
+```bibtex
+@inproceedings{rafailov2023direct,
+    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
+    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
+    year         = 2023,
+    booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
+    url          = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
+    editor       = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
+}
+```
+
+Cite TRL as:
+    
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

all_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 0.9971988795518207,
+    "total_flos": 0.0,
+    "train_loss": 0.47946612754564605,
+    "train_runtime": 42499.5655,
+    "train_samples": 45608,
+    "train_samples_per_second": 1.073,
+    "train_steps_per_second": 0.004
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.46.2"
+}

logs/amlt_code_runner.txt

@@ -1,22 +1,22 @@
-2025-03-01 05:51:32,812:amlt-code-runner:INFO - SINGULARITY_LOCATION: westus2
-2025-03-01 05:51:32,812:amlt-code-runner:INFO - AISC_INSTANCE_TYPE: Singularity.NC96ad_A100_v4
-2025-03-01 05:51:34,063:amlt-code-runner:INFO - Not removing AzureML's cd commands from /etc/profile due to an error: [Errno 13] Permission denied: '/etc/profile'
-2025-03-01 05:51:34,063:amlt-code-runner:WARNING - Environment variable 'NCCL_SOCKET_IFNAME' already set to 'eth0', not changing to '^docker0,lo'
-2025-03-01 05:51:34,063:amlt-code-runner:INFO - RANK = 0
-2025-03-01 05:51:34,063:amlt-code-runner:INFO - LOCAL_RANK = None
-2025-03-01 05:51:34,064:amlt-code-runner:INFO - WORLD_SIZE = 1
-2025-03-01 05:51:34,064:amlt-code-runner:INFO - MASTER_ADDR = node-0
-2025-03-01 05:51:34,064:amlt-code-runner:INFO - MASTER_PORT = 9500
-2025-03-01 05:51:34,065:amlt-code-runner:WARNING - Installing amlt runtime dependencies: ['wrapt', 'azure-identity', 'python-dateutil', 'pytz'] into /tmp/amlt-user-base
-2025-03-01 05:51:35,296:amlt-code-runner:INFO - Setting WANDB_RUN_ID to 'kind_onion_8sfvnlnwfk_9'
-2025-03-01 05:51:35,296:amlt-code-runner:INFO - Expanding HyperDrive arguments into /tmp/amlt_run_hd.sh
-2025-03-01 05:51:35,592:amlt-code-runner:INFO - Parsing tracking uri /mlflow/v1.0/subscriptions/2aac527a-de5a-4fe3-95e9-5c8b9d48ed62/resourceGroups/cyrilzhang/providers/Microsoft.MachineLearningServices/workspaces/cyrilzhangws
-2025-03-01 05:51:35,592:amlt-code-runner:INFO - Tracking uri /mlflow/v1.0/subscriptions/2aac527a-de5a-4fe3-95e9-5c8b9d48ed62/resourceGroups/cyrilzhang/providers/Microsoft.MachineLearningServices/workspaces/cyrilzhangws has sub id 2aac527a-de5a-4fe3-95e9-5c8b9d48ed62, resource group cyrilzhang, and workspace cyrilzhangws
-2025-03-01 05:51:35,592:aml_token_auth:WARNING - The AzureMLTokenAuthentication created will not be updated due to missing params. The token expires on 2025-03-20 19:29:14.
+2025-02-28 18:16:48,809:amlt-code-runner:INFO - SINGULARITY_LOCATION: westus2
+2025-02-28 18:16:48,809:amlt-code-runner:INFO - AISC_INSTANCE_TYPE: Singularity.NC96ad_A100_v4
+2025-02-28 18:16:50,130:amlt-code-runner:INFO - Not removing AzureML's cd commands from /etc/profile due to an error: [Errno 13] Permission denied: '/etc/profile'
+2025-02-28 18:16:50,130:amlt-code-runner:WARNING - Environment variable 'NCCL_SOCKET_IFNAME' already set to 'eth0', not changing to '^docker0,lo'
+2025-02-28 18:16:50,130:amlt-code-runner:INFO - RANK = 0
+2025-02-28 18:16:50,130:amlt-code-runner:INFO - LOCAL_RANK = None
+2025-02-28 18:16:50,130:amlt-code-runner:INFO - WORLD_SIZE = 1
+2025-02-28 18:16:50,131:amlt-code-runner:INFO - MASTER_ADDR = node-0
+2025-02-28 18:16:50,131:amlt-code-runner:INFO - MASTER_PORT = 9500
+2025-02-28 18:16:50,132:amlt-code-runner:WARNING - Installing amlt runtime dependencies: ['wrapt', 'azure-identity', 'python-dateutil', 'pytz'] into /tmp/amlt-user-base
+2025-02-28 18:16:51,378:amlt-code-runner:INFO - Setting WANDB_RUN_ID to 'kind_onion_8sfvnlnwfk_1'
+2025-02-28 18:16:51,378:amlt-code-runner:INFO - Expanding HyperDrive arguments into /tmp/amlt_run_hd.sh
+2025-02-28 18:16:51,760:amlt-code-runner:INFO - Parsing tracking uri /mlflow/v1.0/subscriptions/2aac527a-de5a-4fe3-95e9-5c8b9d48ed62/resourceGroups/cyrilzhang/providers/Microsoft.MachineLearningServices/workspaces/cyrilzhangws
+2025-02-28 18:16:51,760:amlt-code-runner:INFO - Tracking uri /mlflow/v1.0/subscriptions/2aac527a-de5a-4fe3-95e9-5c8b9d48ed62/resourceGroups/cyrilzhang/providers/Microsoft.MachineLearningServices/workspaces/cyrilzhangws has sub id 2aac527a-de5a-4fe3-95e9-5c8b9d48ed62, resource group cyrilzhang, and workspace cyrilzhangws
+2025-02-28 18:16:51,761:aml_token_auth:WARNING - The AzureMLTokenAuthentication created will not be updated due to missing params. The token expires on 2025-03-20 19:29:14.
 
-2025-03-01 05:51:35,594:urllib3.connectionpool:DEBUG - Starting new HTTPS connection (1): eastus.api.azureml.ms:443
-2025-03-01 05:51:35,959:urllib3.connectionpool:DEBUG - https://eastus.api.azureml.ms:443 "POST /mlflow/v1.0/subscriptions/2aac527a-de5a-4fe3-95e9-5c8b9d48ed62/resourceGroups/cyrilzhang/providers/Microsoft.MachineLearningServices/workspaces/cyrilzhangws/api/2.0/mlflow/runs/set-tag HTTP/11" 403 0
-2025-03-01 05:51:35,960:amlt-code-runner:WARNING - Failed to rename job according to the amulet job name template. Run 'amlt list' client side to set the display name according to the amulet job template name. The error we encountered was: Failed to update display name: 
-2025-03-01 05:51:35,982:amlt-code-runner:INFO - Executing ./amlt_setup.sh, /tmp/amlt_run_hd.sh
-2025-03-01 05:51:36,053:background_dirsync:INFO - Starting directory syncer from '/scratch/amlt_code/outputs' to '/mnt/output/projects/lmpref/amlt-results/kind_onion_8sfvnlnwfk_9', every 30.000000s
-2025-03-01 05:51:36,056:background_dirsync:INFO - Starting directory syncer from '/scratch/azureml/cr/j/b2202e217c194ab682d6c0de1367ef62/exe/wd/logs' to '/scratch/amlt_code/outputs/logs', every 30.000000s
+2025-02-28 18:16:51,763:urllib3.connectionpool:DEBUG - Starting new HTTPS connection (1): eastus.api.azureml.ms:443
+2025-02-28 18:16:52,103:urllib3.connectionpool:DEBUG - https://eastus.api.azureml.ms:443 "POST /mlflow/v1.0/subscriptions/2aac527a-de5a-4fe3-95e9-5c8b9d48ed62/resourceGroups/cyrilzhang/providers/Microsoft.MachineLearningServices/workspaces/cyrilzhangws/api/2.0/mlflow/runs/set-tag HTTP/11" 403 0
+2025-02-28 18:16:52,104:amlt-code-runner:WARNING - Failed to rename job according to the amulet job name template. Run 'amlt list' client side to set the display name according to the amulet job template name. The error we encountered was: Failed to update display name: 
+2025-02-28 18:16:52,129:amlt-code-runner:INFO - Executing ./amlt_setup.sh, /tmp/amlt_run_hd.sh
+2025-02-28 18:16:52,205:background_dirsync:INFO - Starting directory syncer from '/scratch/amlt_code/outputs' to '/mnt/output/projects/lmpref/amlt-results/kind_onion_8sfvnlnwfk_1', every 30.000000s
+2025-02-28 18:16:52,209:background_dirsync:INFO - Starting directory syncer from '/scratch/azureml/cr/j/a0229e0c1f104241bee93cad752bc62b/exe/wd/logs' to '/scratch/amlt_code/outputs/logs', every 30.000000s

model-00001-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d46fa09e544111a970d75a4b61d649ed92c4fbdd2021b19394a28b3a064446da
+oid sha256:90e289ce017f59f871c3d51c1c8b7e539f37d7325c445f5795432a79a3f165af
 size 4943162336

model-00002-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c9b8f8018f208f2bbb5f2fd725561caf208dacabd15febd219bfae8af6f949d0
+oid sha256:a1dc2ac6bdc0a077eca365dc0856f95a0712043b9956cd920706a6eb0c0a8fef
 size 4999819336

model-00003-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:309866788c8b07654fa0c6316f2a63ac75ca4ce6ecefc11776d8def4d66a6b52
+oid sha256:9654382ccdb09228edad296504657752d1f82348a1c02d2bbaae0211a2ba1018
 size 4540516344

train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 0.9971988795518207,
+    "total_flos": 0.0,
+    "train_loss": 0.47946612754564605,
+    "train_runtime": 42499.5655,
+    "train_samples": 45608,
+    "train_samples_per_second": 1.073,
+    "train_steps_per_second": 0.004
+}

trainer_state.json

@@ -0,0 +1,1736 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9971988795518207,
+  "eval_steps": 2,
+  "global_step": 178,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0056022408963585435,
+      "grad_norm": 7.673953506214025,
+      "learning_rate": 2.7777777777777774e-08,
+      "logits/chosen": -3.109375,
+      "logits/rejected": -3.1875,
+      "logps/chosen": -302.0,
+      "logps/rejected": -274.0,
+      "loss": 0.6914,
+      "rewards/accuracies": 0.0,
+      "rewards/chosen": 0.0,
+      "rewards/margins": 0.0,
+      "rewards/rejected": 0.0,
+      "step": 1
+    },
+    {
+      "epoch": 0.011204481792717087,
+      "eval_logits/chosen": -3.171875,
+      "eval_logits/rejected": -3.21875,
+      "eval_logps/chosen": -342.0,
+      "eval_logps/rejected": -284.0,
+      "eval_loss": 0.6919978857040405,
+      "eval_rewards/accuracies": 0.2239583283662796,
+      "eval_rewards/chosen": -0.0001773834228515625,
+      "eval_rewards/margins": -0.000461578369140625,
+      "eval_rewards/rejected": 0.000286102294921875,
+      "eval_runtime": 137.3657,
+      "eval_samples_per_second": 10.825,
+      "eval_steps_per_second": 0.175,
+      "step": 2
+    },
+    {
+      "epoch": 0.022408963585434174,
+      "eval_logits/chosen": -3.171875,
+      "eval_logits/rejected": -3.21875,
+      "eval_logps/chosen": -342.0,
+      "eval_logps/rejected": -284.0,
+      "eval_loss": 0.692165195941925,
+      "eval_rewards/accuracies": 0.2213541716337204,
+      "eval_rewards/chosen": -0.000286102294921875,
+      "eval_rewards/margins": -0.0003795623779296875,
+      "eval_rewards/rejected": 9.441375732421875e-05,
+      "eval_runtime": 138.4841,
+      "eval_samples_per_second": 10.738,
+      "eval_steps_per_second": 0.173,
+      "step": 4
+    },
+    {
+      "epoch": 0.03361344537815126,
+      "eval_logits/chosen": -3.171875,
+      "eval_logits/rejected": -3.21875,
+      "eval_logps/chosen": -342.0,
+      "eval_logps/rejected": -284.0,
+      "eval_loss": 0.691789448261261,
+      "eval_rewards/accuracies": 0.2630208432674408,
+      "eval_rewards/chosen": 0.000720977783203125,
+      "eval_rewards/margins": 0.00083160400390625,
+      "eval_rewards/rejected": -0.00010824203491210938,
+      "eval_runtime": 139.1361,
+      "eval_samples_per_second": 10.687,
+      "eval_steps_per_second": 0.172,
+      "step": 6
+    },
+    {
+      "epoch": 0.04481792717086835,
+      "eval_logits/chosen": -3.15625,
+      "eval_logits/rejected": -3.21875,
+      "eval_logps/chosen": -342.0,
+      "eval_logps/rejected": -284.0,
+      "eval_loss": 0.6911332011222839,
+      "eval_rewards/accuracies": 0.3229166567325592,
+      "eval_rewards/chosen": 0.002105712890625,
+      "eval_rewards/margins": 0.0025634765625,
+      "eval_rewards/rejected": -0.0004596710205078125,
+      "eval_runtime": 139.4929,
+      "eval_samples_per_second": 10.66,
+      "eval_steps_per_second": 0.172,
+      "step": 8
+    },
+    {
+      "epoch": 0.056022408963585436,
+      "grad_norm": 6.959076756010086,
+      "learning_rate": 2.7777777777777776e-07,
+      "logits/chosen": -3.1875,
+      "logits/rejected": -3.203125,
+      "logps/chosen": -304.0,
+      "logps/rejected": -278.0,
+      "loss": 0.6916,
+      "rewards/accuracies": 0.2361111044883728,
+      "rewards/chosen": 0.00077056884765625,
+      "rewards/margins": 0.00125885009765625,
+      "rewards/rejected": -0.000492095947265625,
+      "step": 10
+    },
+    {
+      "epoch": 0.056022408963585436,
+      "eval_logits/chosen": -3.15625,
+      "eval_logits/rejected": -3.203125,
+      "eval_logps/chosen": -342.0,
+      "eval_logps/rejected": -284.0,
+      "eval_loss": 0.6895715594291687,
+      "eval_rewards/accuracies": 0.515625,
+      "eval_rewards/chosen": 0.00518798828125,
+      "eval_rewards/margins": 0.006134033203125,
+      "eval_rewards/rejected": -0.000934600830078125,
+      "eval_runtime": 139.1903,
+      "eval_samples_per_second": 10.683,
+      "eval_steps_per_second": 0.172,
+      "step": 10
+    },
+    {
+      "epoch": 0.06722689075630252,
+      "eval_logits/chosen": -3.15625,
+      "eval_logits/rejected": -3.203125,
+      "eval_logps/chosen": -340.0,
+      "eval_logps/rejected": -284.0,
+      "eval_loss": 0.6875709891319275,
+      "eval_rewards/accuracies": 0.6119791865348816,
+      "eval_rewards/chosen": 0.00970458984375,
+      "eval_rewards/margins": 0.01190185546875,
+      "eval_rewards/rejected": -0.002227783203125,
+      "eval_runtime": 138.8331,
+      "eval_samples_per_second": 10.711,
+      "eval_steps_per_second": 0.173,
+      "step": 12
+    },
+    {
+      "epoch": 0.0784313725490196,
+      "eval_logits/chosen": -3.140625,
+      "eval_logits/rejected": -3.1875,
+      "eval_logps/chosen": -340.0,
+      "eval_logps/rejected": -284.0,
+      "eval_loss": 0.6827700138092041,
+      "eval_rewards/accuracies": 0.6875,
+      "eval_rewards/chosen": 0.01519775390625,
+      "eval_rewards/margins": 0.0230712890625,
+      "eval_rewards/rejected": -0.00787353515625,
+      "eval_runtime": 138.8101,
+      "eval_samples_per_second": 10.712,
+      "eval_steps_per_second": 0.173,
+      "step": 14
+    },
+    {
+      "epoch": 0.0896358543417367,
+      "eval_logits/chosen": -3.140625,
+      "eval_logits/rejected": -3.1875,
+      "eval_logps/chosen": -340.0,
+      "eval_logps/rejected": -284.0,
+      "eval_loss": 0.6777191162109375,
+      "eval_rewards/accuracies": 0.7239583134651184,
+      "eval_rewards/chosen": 0.018798828125,
+      "eval_rewards/margins": 0.03564453125,
+      "eval_rewards/rejected": -0.0167236328125,
+      "eval_runtime": 138.2876,
+      "eval_samples_per_second": 10.753,
+      "eval_steps_per_second": 0.174,
+      "step": 16
+    },
+    {
+      "epoch": 0.10084033613445378,
+      "eval_logits/chosen": -3.125,
+      "eval_logits/rejected": -3.171875,
+      "eval_logps/chosen": -340.0,
+      "eval_logps/rejected": -286.0,
+      "eval_loss": 0.6712060570716858,
+      "eval_rewards/accuracies": 0.7526041865348816,
+      "eval_rewards/chosen": 0.0186767578125,
+      "eval_rewards/margins": 0.051513671875,
+      "eval_rewards/rejected": -0.032958984375,
+      "eval_runtime": 138.2305,
+      "eval_samples_per_second": 10.757,
+      "eval_steps_per_second": 0.174,
+      "step": 18
+    },
+    {
+      "epoch": 0.11204481792717087,
+      "grad_norm": 7.180511227020638,
+      "learning_rate": 4.998072590601808e-07,
+      "logits/chosen": -3.140625,
+      "logits/rejected": -3.171875,
+      "logps/chosen": -324.0,
+      "logps/rejected": -278.0,
+      "loss": 0.6803,
+      "rewards/accuracies": 0.6484375,
+      "rewards/chosen": 0.01397705078125,
+      "rewards/margins": 0.0272216796875,
+      "rewards/rejected": -0.01318359375,
+      "step": 20
+    },
+    {
+      "epoch": 0.11204481792717087,
+      "eval_logits/chosen": -3.09375,
+      "eval_logits/rejected": -3.140625,
+      "eval_logps/chosen": -340.0,
+      "eval_logps/rejected": -290.0,
+      "eval_loss": 0.6606504917144775,
+      "eval_rewards/accuracies": 0.7682291865348816,
+      "eval_rewards/chosen": 0.01080322265625,
+      "eval_rewards/margins": 0.078125,
+      "eval_rewards/rejected": -0.0673828125,
+      "eval_runtime": 138.5889,
+      "eval_samples_per_second": 10.73,
+      "eval_steps_per_second": 0.173,
+      "step": 20
+    },
+    {
+      "epoch": 0.12324929971988796,
+      "eval_logits/chosen": -3.078125,
+      "eval_logits/rejected": -3.125,
+      "eval_logps/chosen": -342.0,
+      "eval_logps/rejected": -294.0,
+      "eval_loss": 0.650039792060852,
+      "eval_rewards/accuracies": 0.7734375,
+      "eval_rewards/chosen": -0.0037689208984375,
+      "eval_rewards/margins": 0.1083984375,
+      "eval_rewards/rejected": -0.1123046875,
+      "eval_runtime": 139.3628,
+      "eval_samples_per_second": 10.67,
+      "eval_steps_per_second": 0.172,
+      "step": 22
+    },
+    {
+      "epoch": 0.13445378151260504,
+      "eval_logits/chosen": -3.0625,
+      "eval_logits/rejected": -3.109375,
+      "eval_logps/chosen": -344.0,
+      "eval_logps/rejected": -300.0,
+      "eval_loss": 0.6396481990814209,
+      "eval_rewards/accuracies": 0.78125,
+      "eval_rewards/chosen": -0.0281982421875,
+      "eval_rewards/margins": 0.138671875,
+      "eval_rewards/rejected": -0.1669921875,
+      "eval_runtime": 138.9259,
+      "eval_samples_per_second": 10.704,
+      "eval_steps_per_second": 0.173,
+      "step": 24
+    },
+    {
+      "epoch": 0.14565826330532214,
+      "eval_logits/chosen": -3.0625,
+      "eval_logits/rejected": -3.109375,
+      "eval_logps/chosen": -348.0,
+      "eval_logps/rejected": -308.0,
+      "eval_loss": 0.6289721727371216,
+      "eval_rewards/accuracies": 0.7760416865348816,
+      "eval_rewards/chosen": -0.06787109375,
+      "eval_rewards/margins": 0.171875,
+      "eval_rewards/rejected": -0.240234375,
+      "eval_runtime": 138.2868,
+      "eval_samples_per_second": 10.753,
+      "eval_steps_per_second": 0.174,
+      "step": 26
+    },
+    {
+      "epoch": 0.1568627450980392,
+      "eval_logits/chosen": -3.046875,
+      "eval_logits/rejected": -3.09375,
+      "eval_logps/chosen": -354.0,
+      "eval_logps/rejected": -316.0,
+      "eval_loss": 0.6160576939582825,
+      "eval_rewards/accuracies": 0.7760416865348816,
+      "eval_rewards/chosen": -0.1123046875,
+      "eval_rewards/margins": 0.2138671875,
+      "eval_rewards/rejected": -0.326171875,
+      "eval_runtime": 138.8043,
+      "eval_samples_per_second": 10.713,
+      "eval_steps_per_second": 0.173,
+      "step": 28
+    },
+    {
+      "epoch": 0.16806722689075632,
+      "grad_norm": 6.538297755173619,
+      "learning_rate": 4.930924800994191e-07,
+      "logits/chosen": -3.078125,
+      "logits/rejected": -3.140625,
+      "logps/chosen": -344.0,
+      "logps/rejected": -284.0,
+      "loss": 0.6322,
+      "rewards/accuracies": 0.776562511920929,
+      "rewards/chosen": -0.03662109375,
+      "rewards/margins": 0.15234375,
+      "rewards/rejected": -0.189453125,
+      "step": 30
+    },
+    {
+      "epoch": 0.16806722689075632,
+      "eval_logits/chosen": -3.046875,
+      "eval_logits/rejected": -3.09375,
+      "eval_logps/chosen": -356.0,
+      "eval_logps/rejected": -324.0,
+      "eval_loss": 0.6028054356575012,
+      "eval_rewards/accuracies": 0.78125,
+      "eval_rewards/chosen": -0.1513671875,
+      "eval_rewards/margins": 0.26171875,
+      "eval_rewards/rejected": -0.412109375,
+      "eval_runtime": 138.2396,
+      "eval_samples_per_second": 10.757,
+      "eval_steps_per_second": 0.174,
+      "step": 30
+    },
+    {
+      "epoch": 0.1792717086834734,
+      "eval_logits/chosen": -3.03125,
+      "eval_logits/rejected": -3.078125,
+      "eval_logps/chosen": -360.0,
+      "eval_logps/rejected": -332.0,
+      "eval_loss": 0.5906229615211487,
+      "eval_rewards/accuracies": 0.7890625,
+      "eval_rewards/chosen": -0.1787109375,
+      "eval_rewards/margins": 0.30859375,
+      "eval_rewards/rejected": -0.48828125,
+      "eval_runtime": 139.1647,
+      "eval_samples_per_second": 10.685,
+      "eval_steps_per_second": 0.172,
+      "step": 32
+    },
+    {
+      "epoch": 0.19047619047619047,
+      "eval_logits/chosen": -3.03125,
+      "eval_logits/rejected": -3.078125,
+      "eval_logps/chosen": -362.0,
+      "eval_logps/rejected": -338.0,
+      "eval_loss": 0.5809845924377441,
+      "eval_rewards/accuracies": 0.7838541865348816,
+      "eval_rewards/chosen": -0.2001953125,
+      "eval_rewards/margins": 0.3515625,
+      "eval_rewards/rejected": -0.55078125,
+      "eval_runtime": 138.5178,
+      "eval_samples_per_second": 10.735,
+      "eval_steps_per_second": 0.173,
+      "step": 34
+    },
+    {
+      "epoch": 0.20168067226890757,
+      "eval_logits/chosen": -3.03125,
+      "eval_logits/rejected": -3.0625,
+      "eval_logps/chosen": -366.0,
+      "eval_logps/rejected": -348.0,
+      "eval_loss": 0.570867657661438,
+      "eval_rewards/accuracies": 0.7838541865348816,
+      "eval_rewards/chosen": -0.2490234375,
+      "eval_rewards/margins": 0.404296875,
+      "eval_rewards/rejected": -0.65234375,
+      "eval_runtime": 138.3703,
+      "eval_samples_per_second": 10.747,
+      "eval_steps_per_second": 0.173,
+      "step": 36
+    },
+    {
+      "epoch": 0.21288515406162464,
+      "eval_logits/chosen": -3.03125,
+      "eval_logits/rejected": -3.0625,
+      "eval_logps/chosen": -380.0,
+      "eval_logps/rejected": -368.0,
+      "eval_loss": 0.5650723576545715,
+      "eval_rewards/accuracies": 0.7630208134651184,
+      "eval_rewards/chosen": -0.390625,
+      "eval_rewards/margins": 0.45703125,
+      "eval_rewards/rejected": -0.84765625,
+      "eval_runtime": 139.1352,
+      "eval_samples_per_second": 10.687,
+      "eval_steps_per_second": 0.172,
+      "step": 38
+    },
+    {
+      "epoch": 0.22408963585434175,
+      "grad_norm": 11.8331099493001,
+      "learning_rate": 4.770357934562704e-07,
+      "logits/chosen": -3.015625,
+      "logits/rejected": -3.0,
+      "logps/chosen": -336.0,
+      "logps/rejected": -334.0,
+      "loss": 0.5799,
+      "rewards/accuracies": 0.770312488079071,
+      "rewards/chosen": -0.251953125,
+      "rewards/margins": 0.3359375,
+      "rewards/rejected": -0.58984375,
+      "step": 40
+    },
+    {
+      "epoch": 0.22408963585434175,
+      "eval_logits/chosen": -3.015625,
+      "eval_logits/rejected": -3.0625,
+      "eval_logps/chosen": -380.0,
+      "eval_logps/rejected": -372.0,
+      "eval_loss": 0.555812418460846,
+      "eval_rewards/accuracies": 0.78125,
+      "eval_rewards/chosen": -0.388671875,
+      "eval_rewards/margins": 0.498046875,
+      "eval_rewards/rejected": -0.88671875,
+      "eval_runtime": 138.52,
+      "eval_samples_per_second": 10.735,
+      "eval_steps_per_second": 0.173,
+      "step": 40
+    },
+    {
+      "epoch": 0.23529411764705882,
+      "eval_logits/chosen": -3.015625,
+      "eval_logits/rejected": -3.0625,
+      "eval_logps/chosen": -376.0,
+      "eval_logps/rejected": -370.0,
+      "eval_loss": 0.550049364566803,
+      "eval_rewards/accuracies": 0.7786458134651184,
+      "eval_rewards/chosen": -0.341796875,
+      "eval_rewards/margins": 0.53125,
+      "eval_rewards/rejected": -0.875,
+      "eval_runtime": 138.2263,
+      "eval_samples_per_second": 10.758,
+      "eval_steps_per_second": 0.174,
+      "step": 42
+    },
+    {
+      "epoch": 0.24649859943977592,
+      "eval_logits/chosen": -3.015625,
+      "eval_logits/rejected": -3.046875,
+      "eval_logps/chosen": -376.0,
+      "eval_logps/rejected": -374.0,
+      "eval_loss": 0.5448747277259827,
+      "eval_rewards/accuracies": 0.7838541865348816,
+      "eval_rewards/chosen": -0.349609375,
+      "eval_rewards/margins": 0.5625,
+      "eval_rewards/rejected": -0.91015625,
+      "eval_runtime": 138.6453,
+      "eval_samples_per_second": 10.725,
+      "eval_steps_per_second": 0.173,
+      "step": 44
+    },
+    {
+      "epoch": 0.25770308123249297,
+      "eval_logits/chosen": -3.015625,
+      "eval_logits/rejected": -3.046875,
+      "eval_logps/chosen": -384.0,
+      "eval_logps/rejected": -386.0,
+      "eval_loss": 0.5377079248428345,
+      "eval_rewards/accuracies": 0.7760416865348816,
+      "eval_rewards/chosen": -0.423828125,
+      "eval_rewards/margins": 0.60546875,
+      "eval_rewards/rejected": -1.0234375,
+      "eval_runtime": 138.7181,
+      "eval_samples_per_second": 10.72,
+      "eval_steps_per_second": 0.173,
+      "step": 46
+    },
+    {
+      "epoch": 0.2689075630252101,
+      "eval_logits/chosen": -3.0,
+      "eval_logits/rejected": -3.03125,
+      "eval_logps/chosen": -394.0,
+      "eval_logps/rejected": -398.0,
+      "eval_loss": 0.533993661403656,
+      "eval_rewards/accuracies": 0.7786458134651184,
+      "eval_rewards/chosen": -0.51953125,
+      "eval_rewards/margins": 0.6328125,
+      "eval_rewards/rejected": -1.15625,
+      "eval_runtime": 138.3153,
+      "eval_samples_per_second": 10.751,
+      "eval_steps_per_second": 0.174,
+      "step": 48
+    },
+    {
+      "epoch": 0.2801120448179272,
+      "grad_norm": 11.663036285172442,
+      "learning_rate": 4.5225424859373684e-07,
+      "logits/chosen": -3.0,
+      "logits/rejected": -3.0625,
+      "logps/chosen": -368.0,
+      "logps/rejected": -384.0,
+      "loss": 0.5375,
+      "rewards/accuracies": 0.800000011920929,
+      "rewards/chosen": -0.376953125,
+      "rewards/margins": 0.61328125,
+      "rewards/rejected": -0.9921875,
+      "step": 50
+    },
+    {
+      "epoch": 0.2801120448179272,
+      "eval_logits/chosen": -3.0,
+      "eval_logits/rejected": -3.03125,
+      "eval_logps/chosen": -392.0,
+      "eval_logps/rejected": -400.0,
+      "eval_loss": 0.5259566307067871,
+      "eval_rewards/accuracies": 0.7916666865348816,
+      "eval_rewards/chosen": -0.4921875,
+      "eval_rewards/margins": 0.66796875,
+      "eval_rewards/rejected": -1.15625,
+      "eval_runtime": 138.3491,
+      "eval_samples_per_second": 10.748,
+      "eval_steps_per_second": 0.173,
+      "step": 50
+    },
+    {
+      "epoch": 0.2913165266106443,
+      "eval_logits/chosen": -2.984375,
+      "eval_logits/rejected": -3.015625,
+      "eval_logps/chosen": -386.0,
+      "eval_logps/rejected": -400.0,
+      "eval_loss": 0.5105913877487183,
+      "eval_rewards/accuracies": 0.796875,
+      "eval_rewards/chosen": -0.44921875,
+      "eval_rewards/margins": 0.71484375,
+      "eval_rewards/rejected": -1.1640625,
+      "eval_runtime": 138.2221,
+      "eval_samples_per_second": 10.758,
+      "eval_steps_per_second": 0.174,
+      "step": 52
+    },
+    {
+      "epoch": 0.3025210084033613,
+      "eval_logits/chosen": -2.96875,
+      "eval_logits/rejected": -3.0,
+      "eval_logps/chosen": -396.0,
+      "eval_logps/rejected": -416.0,
+      "eval_loss": 0.4958760142326355,
+      "eval_rewards/accuracies": 0.8098958134651184,
+      "eval_rewards/chosen": -0.54296875,
+      "eval_rewards/margins": 0.7890625,
+      "eval_rewards/rejected": -1.328125,
+      "eval_runtime": 139.0933,
+      "eval_samples_per_second": 10.691,
+      "eval_steps_per_second": 0.173,
+      "step": 54
+    },
+    {
+      "epoch": 0.3137254901960784,
+      "eval_logits/chosen": -2.953125,
+      "eval_logits/rejected": -2.96875,
+      "eval_logps/chosen": -416.0,
+      "eval_logps/rejected": -448.0,
+      "eval_loss": 0.48305684328079224,
+      "eval_rewards/accuracies": 0.8046875,
+      "eval_rewards/chosen": -0.75,
+      "eval_rewards/margins": 0.890625,
+      "eval_rewards/rejected": -1.640625,
+      "eval_runtime": 138.1956,
+      "eval_samples_per_second": 10.76,
+      "eval_steps_per_second": 0.174,
+      "step": 56
+    },
+    {
+      "epoch": 0.32492997198879553,
+      "eval_logits/chosen": -2.921875,
+      "eval_logits/rejected": -2.953125,
+      "eval_logps/chosen": -434.0,
+      "eval_logps/rejected": -470.0,
+      "eval_loss": 0.48011937737464905,
+      "eval_rewards/accuracies": 0.8203125,
+      "eval_rewards/chosen": -0.92578125,
+      "eval_rewards/margins": 0.9453125,
+      "eval_rewards/rejected": -1.8671875,
+      "eval_runtime": 138.1284,
+      "eval_samples_per_second": 10.765,
+      "eval_steps_per_second": 0.174,
+      "step": 58
+    },
+    {
+      "epoch": 0.33613445378151263,
+      "grad_norm": 14.035000063763073,
+      "learning_rate": 4.1970018638323547e-07,
+      "logits/chosen": -3.03125,
+      "logits/rejected": -3.046875,
+      "logps/chosen": -386.0,
+      "logps/rejected": -410.0,
+      "loss": 0.4943,
+      "rewards/accuracies": 0.824999988079071,
+      "rewards/chosen": -0.61328125,
+      "rewards/margins": 0.83984375,
+      "rewards/rejected": -1.453125,
+      "step": 60
+    },
+    {
+      "epoch": 0.33613445378151263,
+      "eval_logits/chosen": -2.90625,
+      "eval_logits/rejected": -2.921875,
+      "eval_logps/chosen": -428.0,
+      "eval_logps/rejected": -470.0,
+      "eval_loss": 0.4731064438819885,
+      "eval_rewards/accuracies": 0.8333333134651184,
+      "eval_rewards/chosen": -0.87109375,
+      "eval_rewards/margins": 1.0,
+      "eval_rewards/rejected": -1.875,
+      "eval_runtime": 139.0999,
+      "eval_samples_per_second": 10.69,
+      "eval_steps_per_second": 0.173,
+      "step": 60
+    },
+    {
+      "epoch": 0.3473389355742297,
+      "eval_logits/chosen": -2.875,
+      "eval_logits/rejected": -2.890625,
+      "eval_logps/chosen": -420.0,
+      "eval_logps/rejected": -466.0,
+      "eval_loss": 0.46384912729263306,
+      "eval_rewards/accuracies": 0.8255208134651184,
+      "eval_rewards/chosen": -0.78515625,
+      "eval_rewards/margins": 1.046875,
+      "eval_rewards/rejected": -1.8359375,
+      "eval_runtime": 138.3433,
+      "eval_samples_per_second": 10.749,
+      "eval_steps_per_second": 0.173,
+      "step": 62
+    },
+    {
+      "epoch": 0.3585434173669468,
+      "eval_logits/chosen": -2.828125,
+      "eval_logits/rejected": -2.859375,
+      "eval_logps/chosen": -420.0,
+      "eval_logps/rejected": -470.0,
+      "eval_loss": 0.4573279321193695,
+      "eval_rewards/accuracies": 0.8333333134651184,
+      "eval_rewards/chosen": -0.78125,
+      "eval_rewards/margins": 1.09375,
+      "eval_rewards/rejected": -1.875,
+      "eval_runtime": 139.6746,
+      "eval_samples_per_second": 10.646,
+      "eval_steps_per_second": 0.172,
+      "step": 64
+    },
+    {
+      "epoch": 0.3697478991596639,
+      "eval_logits/chosen": -2.796875,
+      "eval_logits/rejected": -2.828125,
+      "eval_logps/chosen": -416.0,
+      "eval_logps/rejected": -470.0,
+      "eval_loss": 0.4575417637825012,
+      "eval_rewards/accuracies": 0.828125,
+      "eval_rewards/chosen": -0.73828125,
+      "eval_rewards/margins": 1.125,
+      "eval_rewards/rejected": -1.859375,
+      "eval_runtime": 140.358,
+      "eval_samples_per_second": 10.594,
+      "eval_steps_per_second": 0.171,
+      "step": 66
+    },
+    {
+      "epoch": 0.38095238095238093,
+      "eval_logits/chosen": -2.78125,
+      "eval_logits/rejected": -2.796875,
+      "eval_logps/chosen": -420.0,
+      "eval_logps/rejected": -478.0,
+      "eval_loss": 0.4555843770503998,
+      "eval_rewards/accuracies": 0.8255208134651184,
+      "eval_rewards/chosen": -0.7734375,
+      "eval_rewards/margins": 1.1796875,
+      "eval_rewards/rejected": -1.953125,
+      "eval_runtime": 138.3728,
+      "eval_samples_per_second": 10.746,
+      "eval_steps_per_second": 0.173,
+      "step": 68
+    },
+    {
+      "epoch": 0.39215686274509803,
+      "grad_norm": 13.259815142949064,
+      "learning_rate": 3.806246411789872e-07,
+      "logits/chosen": -2.8125,
+      "logits/rejected": -2.84375,
+      "logps/chosen": -384.0,
+      "logps/rejected": -456.0,
+      "loss": 0.4539,
+      "rewards/accuracies": 0.846875011920929,
+      "rewards/chosen": -0.7578125,
+      "rewards/margins": 1.0859375,
+      "rewards/rejected": -1.84375,
+      "step": 70
+    },
+    {
+      "epoch": 0.39215686274509803,
+      "eval_logits/chosen": -2.765625,
+      "eval_logits/rejected": -2.78125,
+      "eval_logps/chosen": -424.0,
+      "eval_logps/rejected": -488.0,
+      "eval_loss": 0.45154088735580444,
+      "eval_rewards/accuracies": 0.828125,
+      "eval_rewards/chosen": -0.80859375,
+      "eval_rewards/margins": 1.2421875,
+      "eval_rewards/rejected": -2.046875,
+      "eval_runtime": 139.2924,
+      "eval_samples_per_second": 10.675,
+      "eval_steps_per_second": 0.172,
+      "step": 70
+    },
+    {
+      "epoch": 0.40336134453781514,
+      "eval_logits/chosen": -2.765625,
+      "eval_logits/rejected": -2.765625,
+      "eval_logps/chosen": -424.0,
+      "eval_logps/rejected": -492.0,
+      "eval_loss": 0.4482435882091522,
+      "eval_rewards/accuracies": 0.8307291865348816,
+      "eval_rewards/chosen": -0.81640625,
+      "eval_rewards/margins": 1.2734375,
+      "eval_rewards/rejected": -2.09375,
+      "eval_runtime": 139.628,
+      "eval_samples_per_second": 10.65,
+      "eval_steps_per_second": 0.172,
+      "step": 72
+    },
+    {
+      "epoch": 0.41456582633053224,
+      "eval_logits/chosen": -2.734375,
+      "eval_logits/rejected": -2.75,
+      "eval_logps/chosen": -434.0,
+      "eval_logps/rejected": -504.0,
+      "eval_loss": 0.44465625286102295,
+      "eval_rewards/accuracies": 0.8411458134651184,
+      "eval_rewards/chosen": -0.9140625,
+      "eval_rewards/margins": 1.296875,
+      "eval_rewards/rejected": -2.21875,
+      "eval_runtime": 138.4674,
+      "eval_samples_per_second": 10.739,
+      "eval_steps_per_second": 0.173,
+      "step": 74
+    },
+    {
+      "epoch": 0.4257703081232493,
+      "eval_logits/chosen": -2.734375,
+      "eval_logits/rejected": -2.734375,
+      "eval_logps/chosen": -442.0,
+      "eval_logps/rejected": -512.0,
+      "eval_loss": 0.4416819214820862,
+      "eval_rewards/accuracies": 0.8333333134651184,
+      "eval_rewards/chosen": -1.0,
+      "eval_rewards/margins": 1.2890625,
+      "eval_rewards/rejected": -2.296875,
+      "eval_runtime": 138.8776,
+      "eval_samples_per_second": 10.707,
+      "eval_steps_per_second": 0.173,
+      "step": 76
+    },
+    {
+      "epoch": 0.4369747899159664,
+      "eval_logits/chosen": -2.703125,
+      "eval_logits/rejected": -2.703125,
+      "eval_logps/chosen": -438.0,
+      "eval_logps/rejected": -510.0,
+      "eval_loss": 0.43734651803970337,
+      "eval_rewards/accuracies": 0.84375,
+      "eval_rewards/chosen": -0.97265625,
+      "eval_rewards/margins": 1.296875,
+      "eval_rewards/rejected": -2.265625,
+      "eval_runtime": 139.0112,
+      "eval_samples_per_second": 10.697,
+      "eval_steps_per_second": 0.173,
+      "step": 78
+    },
+    {
+      "epoch": 0.4481792717086835,
+      "grad_norm": 13.586202979732352,
+      "learning_rate": 3.3652926426937325e-07,
+      "logits/chosen": -2.78125,
+      "logits/rejected": -2.78125,
+      "logps/chosen": -432.0,
+      "logps/rejected": -496.0,
+      "loss": 0.4342,
+      "rewards/accuracies": 0.840624988079071,
+      "rewards/chosen": -0.9296875,
+      "rewards/margins": 1.2734375,
+      "rewards/rejected": -2.203125,
+      "step": 80
+    },
+    {
+      "epoch": 0.4481792717086835,
+      "eval_logits/chosen": -2.6875,
+      "eval_logits/rejected": -2.6875,
+      "eval_logps/chosen": -438.0,
+      "eval_logps/rejected": -510.0,
+      "eval_loss": 0.4349703788757324,
+      "eval_rewards/accuracies": 0.8541666865348816,
+      "eval_rewards/chosen": -0.97265625,
+      "eval_rewards/margins": 1.296875,
+      "eval_rewards/rejected": -2.265625,
+      "eval_runtime": 138.785,
+      "eval_samples_per_second": 10.714,
+      "eval_steps_per_second": 0.173,
+      "step": 80
+    },
+    {
+      "epoch": 0.45938375350140054,
+      "eval_logits/chosen": -2.671875,
+      "eval_logits/rejected": -2.671875,
+      "eval_logps/chosen": -434.0,
+      "eval_logps/rejected": -504.0,
+      "eval_loss": 0.4350964426994324,
+      "eval_rewards/accuracies": 0.8541666865348816,
+      "eval_rewards/chosen": -0.921875,
+      "eval_rewards/margins": 1.28125,
+      "eval_rewards/rejected": -2.203125,
+      "eval_runtime": 138.9282,
+      "eval_samples_per_second": 10.703,
+      "eval_steps_per_second": 0.173,
+      "step": 82
+    },
+    {
+      "epoch": 0.47058823529411764,
+      "eval_logits/chosen": -2.65625,
+      "eval_logits/rejected": -2.65625,
+      "eval_logps/chosen": -428.0,
+      "eval_logps/rejected": -496.0,
+      "eval_loss": 0.43425795435905457,
+      "eval_rewards/accuracies": 0.8515625,
+      "eval_rewards/chosen": -0.86328125,
+      "eval_rewards/margins": 1.2734375,
+      "eval_rewards/rejected": -2.140625,
+      "eval_runtime": 138.5465,
+      "eval_samples_per_second": 10.733,
+      "eval_steps_per_second": 0.173,
+      "step": 84
+    },
+    {
+      "epoch": 0.48179271708683474,
+      "eval_logits/chosen": -2.640625,
+      "eval_logits/rejected": -2.640625,
+      "eval_logps/chosen": -428.0,
+      "eval_logps/rejected": -498.0,
+      "eval_loss": 0.4317302703857422,
+      "eval_rewards/accuracies": 0.8463541865348816,
+      "eval_rewards/chosen": -0.8671875,
+      "eval_rewards/margins": 1.2734375,
+      "eval_rewards/rejected": -2.140625,
+      "eval_runtime": 138.6705,
+      "eval_samples_per_second": 10.723,
+      "eval_steps_per_second": 0.173,
+      "step": 86
+    },
+    {
+      "epoch": 0.49299719887955185,
+      "eval_logits/chosen": -2.609375,
+      "eval_logits/rejected": -2.609375,
+      "eval_logps/chosen": -436.0,
+      "eval_logps/rejected": -506.0,
+      "eval_loss": 0.43009501695632935,
+      "eval_rewards/accuracies": 0.8463541865348816,
+      "eval_rewards/chosen": -0.93359375,
+      "eval_rewards/margins": 1.28125,
+      "eval_rewards/rejected": -2.21875,
+      "eval_runtime": 138.8488,
+      "eval_samples_per_second": 10.709,
+      "eval_steps_per_second": 0.173,
+      "step": 88
+    },
+    {
+      "epoch": 0.5042016806722689,
+      "grad_norm": 14.25314849341799,
+      "learning_rate": 2.8910861626005773e-07,
+      "logits/chosen": -2.625,
+      "logits/rejected": -2.59375,
+      "logps/chosen": -414.0,
+      "logps/rejected": -488.0,
+      "loss": 0.4313,
+      "rewards/accuracies": 0.8531249761581421,
+      "rewards/chosen": -0.890625,
+      "rewards/margins": 1.2890625,
+      "rewards/rejected": -2.171875,
+      "step": 90
+    },
+    {
+      "epoch": 0.5042016806722689,
+      "eval_logits/chosen": -2.59375,
+      "eval_logits/rejected": -2.578125,
+      "eval_logps/chosen": -436.0,
+      "eval_logps/rejected": -508.0,
+      "eval_loss": 0.4278578460216522,
+      "eval_rewards/accuracies": 0.8541666865348816,
+      "eval_rewards/chosen": -0.9375,
+      "eval_rewards/margins": 1.3203125,
+      "eval_rewards/rejected": -2.25,
+      "eval_runtime": 138.563,
+      "eval_samples_per_second": 10.732,
+      "eval_steps_per_second": 0.173,
+      "step": 90
+    },
+    {
+      "epoch": 0.5154061624649859,
+      "eval_logits/chosen": -2.5625,
+      "eval_logits/rejected": -2.5625,
+      "eval_logps/chosen": -438.0,
+      "eval_logps/rejected": -512.0,
+      "eval_loss": 0.4267123341560364,
+      "eval_rewards/accuracies": 0.8541666865348816,
+      "eval_rewards/chosen": -0.94921875,
+      "eval_rewards/margins": 1.328125,
+      "eval_rewards/rejected": -2.28125,
+      "eval_runtime": 139.5844,
+      "eval_samples_per_second": 10.653,
+      "eval_steps_per_second": 0.172,
+      "step": 92
+    },
+    {
+      "epoch": 0.5266106442577031,
+      "eval_logits/chosen": -2.546875,
+      "eval_logits/rejected": -2.53125,
+      "eval_logps/chosen": -432.0,
+      "eval_logps/rejected": -508.0,
+      "eval_loss": 0.4254372715950012,
+      "eval_rewards/accuracies": 0.8567708134651184,
+      "eval_rewards/chosen": -0.89453125,
+      "eval_rewards/margins": 1.34375,
+      "eval_rewards/rejected": -2.234375,
+      "eval_runtime": 138.3499,
+      "eval_samples_per_second": 10.748,
+      "eval_steps_per_second": 0.173,
+      "step": 94
+    },
+    {
+      "epoch": 0.5378151260504201,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -430.0,
+      "eval_logps/rejected": -506.0,
+      "eval_loss": 0.4240831136703491,
+      "eval_rewards/accuracies": 0.8567708134651184,
+      "eval_rewards/chosen": -0.875,
+      "eval_rewards/margins": 1.359375,
+      "eval_rewards/rejected": -2.234375,
+      "eval_runtime": 139.4588,
+      "eval_samples_per_second": 10.663,
+      "eval_steps_per_second": 0.172,
+      "step": 96
+    },
+    {
+      "epoch": 0.5490196078431373,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -432.0,
+      "eval_logps/rejected": -510.0,
+      "eval_loss": 0.4220282733440399,
+      "eval_rewards/accuracies": 0.859375,
+      "eval_rewards/chosen": -0.890625,
+      "eval_rewards/margins": 1.375,
+      "eval_rewards/rejected": -2.265625,
+      "eval_runtime": 138.912,
+      "eval_samples_per_second": 10.705,
+      "eval_steps_per_second": 0.173,
+      "step": 98
+    },
+    {
+      "epoch": 0.5602240896358543,
+      "grad_norm": 11.44606895069004,
+      "learning_rate": 2.401850460602329e-07,
+      "logits/chosen": -2.578125,
+      "logits/rejected": -2.53125,
+      "logps/chosen": -412.0,
+      "logps/rejected": -498.0,
+      "loss": 0.4211,
+      "rewards/accuracies": 0.8609374761581421,
+      "rewards/chosen": -0.87890625,
+      "rewards/margins": 1.3359375,
+      "rewards/rejected": -2.21875,
+      "step": 100
+    },
+    {
+      "epoch": 0.5602240896358543,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -432.0,
+      "eval_logps/rejected": -512.0,
+      "eval_loss": 0.41982147097587585,
+      "eval_rewards/accuracies": 0.859375,
+      "eval_rewards/chosen": -0.90234375,
+      "eval_rewards/margins": 1.3984375,
+      "eval_rewards/rejected": -2.296875,
+      "eval_runtime": 137.7117,
+      "eval_samples_per_second": 10.798,
+      "eval_steps_per_second": 0.174,
+      "step": 100
+    },
+    {
+      "epoch": 0.5714285714285714,
+      "eval_logits/chosen": -2.546875,
+      "eval_logits/rejected": -2.53125,
+      "eval_logps/chosen": -428.0,
+      "eval_logps/rejected": -510.0,
+      "eval_loss": 0.41878578066825867,
+      "eval_rewards/accuracies": 0.8567708134651184,
+      "eval_rewards/chosen": -0.859375,
+      "eval_rewards/margins": 1.3984375,
+      "eval_rewards/rejected": -2.25,
+      "eval_runtime": 137.9857,
+      "eval_samples_per_second": 10.776,
+      "eval_steps_per_second": 0.174,
+      "step": 102
+    },
+    {
+      "epoch": 0.5826330532212886,
+      "eval_logits/chosen": -2.546875,
+      "eval_logits/rejected": -2.53125,
+      "eval_logps/chosen": -426.0,
+      "eval_logps/rejected": -508.0,
+      "eval_loss": 0.4178510308265686,
+      "eval_rewards/accuracies": 0.8567708134651184,
+      "eval_rewards/chosen": -0.84765625,
+      "eval_rewards/margins": 1.390625,
+      "eval_rewards/rejected": -2.25,
+      "eval_runtime": 138.7271,
+      "eval_samples_per_second": 10.719,
+      "eval_steps_per_second": 0.173,
+      "step": 104
+    },
+    {
+      "epoch": 0.5938375350140056,
+      "eval_logits/chosen": -2.5625,
+      "eval_logits/rejected": -2.53125,
+      "eval_logps/chosen": -428.0,
+      "eval_logps/rejected": -508.0,
+      "eval_loss": 0.4161478281021118,
+      "eval_rewards/accuracies": 0.8541666865348816,
+      "eval_rewards/chosen": -0.85546875,
+      "eval_rewards/margins": 1.3828125,
+      "eval_rewards/rejected": -2.234375,
+      "eval_runtime": 138.8714,
+      "eval_samples_per_second": 10.708,
+      "eval_steps_per_second": 0.173,
+      "step": 106
+    },
+    {
+      "epoch": 0.6050420168067226,
+      "eval_logits/chosen": -2.5625,
+      "eval_logits/rejected": -2.546875,
+      "eval_logps/chosen": -428.0,
+      "eval_logps/rejected": -506.0,
+      "eval_loss": 0.41500023007392883,
+      "eval_rewards/accuracies": 0.8567708134651184,
+      "eval_rewards/chosen": -0.86328125,
+      "eval_rewards/margins": 1.3671875,
+      "eval_rewards/rejected": -2.234375,
+      "eval_runtime": 138.4468,
+      "eval_samples_per_second": 10.741,
+      "eval_steps_per_second": 0.173,
+      "step": 108
+    },
+    {
+      "epoch": 0.6162464985994398,
+      "grad_norm": 12.278538881178221,
+      "learning_rate": 1.9163865903602372e-07,
+      "logits/chosen": -2.578125,
+      "logits/rejected": -2.484375,
+      "logps/chosen": -406.0,
+      "logps/rejected": -512.0,
+      "loss": 0.4106,
+      "rewards/accuracies": 0.878125011920929,
+      "rewards/chosen": -0.875,
+      "rewards/margins": 1.4453125,
+      "rewards/rejected": -2.3125,
+      "step": 110
+    },
+    {
+      "epoch": 0.6162464985994398,
+      "eval_logits/chosen": -2.546875,
+      "eval_logits/rejected": -2.53125,
+      "eval_logps/chosen": -430.0,
+      "eval_logps/rejected": -506.0,
+      "eval_loss": 0.41475099325180054,
+      "eval_rewards/accuracies": 0.8541666865348816,
+      "eval_rewards/chosen": -0.88671875,
+      "eval_rewards/margins": 1.3515625,
+      "eval_rewards/rejected": -2.234375,
+      "eval_runtime": 138.8061,
+      "eval_samples_per_second": 10.713,
+      "eval_steps_per_second": 0.173,
+      "step": 110
+    },
+    {
+      "epoch": 0.6274509803921569,
+      "eval_logits/chosen": -2.546875,
+      "eval_logits/rejected": -2.53125,
+      "eval_logps/chosen": -428.0,
+      "eval_logps/rejected": -502.0,
+      "eval_loss": 0.41493356227874756,
+      "eval_rewards/accuracies": 0.8567708134651184,
+      "eval_rewards/chosen": -0.85546875,
+      "eval_rewards/margins": 1.3359375,
+      "eval_rewards/rejected": -2.1875,
+      "eval_runtime": 139.8268,
+      "eval_samples_per_second": 10.635,
+      "eval_steps_per_second": 0.172,
+      "step": 112
+    },
+    {
+      "epoch": 0.6386554621848739,
+      "eval_logits/chosen": -2.546875,
+      "eval_logits/rejected": -2.53125,
+      "eval_logps/chosen": -428.0,
+      "eval_logps/rejected": -504.0,
+      "eval_loss": 0.41408368945121765,
+      "eval_rewards/accuracies": 0.859375,
+      "eval_rewards/chosen": -0.859375,
+      "eval_rewards/margins": 1.34375,
+      "eval_rewards/rejected": -2.203125,
+      "eval_runtime": 137.7419,
+      "eval_samples_per_second": 10.796,
+      "eval_steps_per_second": 0.174,
+      "step": 114
+    },
+    {
+      "epoch": 0.6498599439775911,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -436.0,
+      "eval_logps/rejected": -512.0,
+      "eval_loss": 0.412348210811615,
+      "eval_rewards/accuracies": 0.8567708134651184,
+      "eval_rewards/chosen": -0.93359375,
+      "eval_rewards/margins": 1.3515625,
+      "eval_rewards/rejected": -2.28125,
+      "eval_runtime": 139.942,
+      "eval_samples_per_second": 10.626,
+      "eval_steps_per_second": 0.171,
+      "step": 116
+    },
+    {
+      "epoch": 0.6610644257703081,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -442.0,
+      "eval_logps/rejected": -520.0,
+      "eval_loss": 0.4106382429599762,
+      "eval_rewards/accuracies": 0.8619791865348816,
+      "eval_rewards/chosen": -1.0078125,
+      "eval_rewards/margins": 1.3828125,
+      "eval_rewards/rejected": -2.390625,
+      "eval_runtime": 138.1581,
+      "eval_samples_per_second": 10.763,
+      "eval_steps_per_second": 0.174,
+      "step": 118
+    },
+    {
+      "epoch": 0.6722689075630253,
+      "grad_norm": 16.28140733426518,
+      "learning_rate": 1.4533506561564305e-07,
+      "logits/chosen": -2.546875,
+      "logits/rejected": -2.5,
+      "logps/chosen": -434.0,
+      "logps/rejected": -520.0,
+      "loss": 0.4259,
+      "rewards/accuracies": 0.856249988079071,
+      "rewards/chosen": -0.9609375,
+      "rewards/margins": 1.296875,
+      "rewards/rejected": -2.265625,
+      "step": 120
+    },
+    {
+      "epoch": 0.6722689075630253,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -440.0,
+      "eval_logps/rejected": -520.0,
+      "eval_loss": 0.4103623032569885,
+      "eval_rewards/accuracies": 0.8645833134651184,
+      "eval_rewards/chosen": -0.984375,
+      "eval_rewards/margins": 1.3984375,
+      "eval_rewards/rejected": -2.375,
+      "eval_runtime": 138.5632,
+      "eval_samples_per_second": 10.732,
+      "eval_steps_per_second": 0.173,
+      "step": 120
+    },
+    {
+      "epoch": 0.6834733893557423,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -438.0,
+      "eval_logps/rejected": -520.0,
+      "eval_loss": 0.41030353307724,
+      "eval_rewards/accuracies": 0.859375,
+      "eval_rewards/chosen": -0.96875,
+      "eval_rewards/margins": 1.4140625,
+      "eval_rewards/rejected": -2.375,
+      "eval_runtime": 138.4762,
+      "eval_samples_per_second": 10.738,
+      "eval_steps_per_second": 0.173,
+      "step": 122
+    },
+    {
+      "epoch": 0.6946778711484594,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -434.0,
+      "eval_logps/rejected": -516.0,
+      "eval_loss": 0.41081422567367554,
+      "eval_rewards/accuracies": 0.8619791865348816,
+      "eval_rewards/chosen": -0.92578125,
+      "eval_rewards/margins": 1.421875,
+      "eval_rewards/rejected": -2.34375,
+      "eval_runtime": 137.8634,
+      "eval_samples_per_second": 10.786,
+      "eval_steps_per_second": 0.174,
+      "step": 124
+    },
+    {
+      "epoch": 0.7058823529411765,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -430.0,
+      "eval_logps/rejected": -512.0,
+      "eval_loss": 0.4114457964897156,
+      "eval_rewards/accuracies": 0.8645833134651184,
+      "eval_rewards/chosen": -0.87890625,
+      "eval_rewards/margins": 1.421875,
+      "eval_rewards/rejected": -2.296875,
+      "eval_runtime": 137.9439,
+      "eval_samples_per_second": 10.78,
+      "eval_steps_per_second": 0.174,
+      "step": 126
+    },
+    {
+      "epoch": 0.7170868347338936,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -432.0,
+      "eval_logps/rejected": -516.0,
+      "eval_loss": 0.41083893179893494,
+      "eval_rewards/accuracies": 0.8697916865348816,
+      "eval_rewards/chosen": -0.90234375,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.328125,
+      "eval_runtime": 138.0714,
+      "eval_samples_per_second": 10.77,
+      "eval_steps_per_second": 0.174,
+      "step": 128
+    },
+    {
+      "epoch": 0.7282913165266106,
+      "grad_norm": 13.33996742784076,
+      "learning_rate": 1.0305368692688174e-07,
+      "logits/chosen": -2.5625,
+      "logits/rejected": -2.515625,
+      "logps/chosen": -416.0,
+      "logps/rejected": -520.0,
+      "loss": 0.4049,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.9609375,
+      "rewards/margins": 1.46875,
+      "rewards/rejected": -2.421875,
+      "step": 130
+    },
+    {
+      "epoch": 0.7282913165266106,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -438.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.4099928140640259,
+      "eval_rewards/accuracies": 0.8697916865348816,
+      "eval_rewards/chosen": -0.953125,
+      "eval_rewards/margins": 1.4375,
+      "eval_rewards/rejected": -2.390625,
+      "eval_runtime": 138.6186,
+      "eval_samples_per_second": 10.727,
+      "eval_steps_per_second": 0.173,
+      "step": 130
+    },
+    {
+      "epoch": 0.7394957983193278,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -444.0,
+      "eval_logps/rejected": -528.0,
+      "eval_loss": 0.409527063369751,
+      "eval_rewards/accuracies": 0.8723958134651184,
+      "eval_rewards/chosen": -1.015625,
+      "eval_rewards/margins": 1.4375,
+      "eval_rewards/rejected": -2.453125,
+      "eval_runtime": 138.1773,
+      "eval_samples_per_second": 10.762,
+      "eval_steps_per_second": 0.174,
+      "step": 132
+    },
+    {
+      "epoch": 0.7507002801120448,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -448.0,
+      "eval_logps/rejected": -532.0,
+      "eval_loss": 0.4095887541770935,
+      "eval_rewards/accuracies": 0.8723958134651184,
+      "eval_rewards/chosen": -1.0546875,
+      "eval_rewards/margins": 1.4453125,
+      "eval_rewards/rejected": -2.5,
+      "eval_runtime": 138.0767,
+      "eval_samples_per_second": 10.769,
+      "eval_steps_per_second": 0.174,
+      "step": 134
+    },
+    {
+      "epoch": 0.7619047619047619,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -446.0,
+      "eval_logps/rejected": -532.0,
+      "eval_loss": 0.4093005955219269,
+      "eval_rewards/accuracies": 0.8723958134651184,
+      "eval_rewards/chosen": -1.046875,
+      "eval_rewards/margins": 1.4453125,
+      "eval_rewards/rejected": -2.484375,
+      "eval_runtime": 137.8685,
+      "eval_samples_per_second": 10.786,
+      "eval_steps_per_second": 0.174,
+      "step": 136
+    },
+    {
+      "epoch": 0.773109243697479,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -442.0,
+      "eval_logps/rejected": -528.0,
+      "eval_loss": 0.40879443287849426,
+      "eval_rewards/accuracies": 0.875,
+      "eval_rewards/chosen": -1.0078125,
+      "eval_rewards/margins": 1.4453125,
+      "eval_rewards/rejected": -2.453125,
+      "eval_runtime": 139.2498,
+      "eval_samples_per_second": 10.679,
+      "eval_steps_per_second": 0.172,
+      "step": 138
+    },
+    {
+      "epoch": 0.7843137254901961,
+      "grad_norm": 15.536009672780233,
+      "learning_rate": 6.641937264107867e-08,
+      "logits/chosen": -2.5625,
+      "logits/rejected": -2.484375,
+      "logps/chosen": -444.0,
+      "logps/rejected": -544.0,
+      "loss": 0.407,
+      "rewards/accuracies": 0.8890625238418579,
+      "rewards/chosen": -1.0546875,
+      "rewards/margins": 1.53125,
+      "rewards/rejected": -2.59375,
+      "step": 140
+    },
+    {
+      "epoch": 0.7843137254901961,
+      "eval_logits/chosen": -2.546875,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -438.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.4085310101509094,
+      "eval_rewards/accuracies": 0.8723958134651184,
+      "eval_rewards/chosen": -0.95703125,
+      "eval_rewards/margins": 1.4375,
+      "eval_rewards/rejected": -2.390625,
+      "eval_runtime": 137.5329,
+      "eval_samples_per_second": 10.812,
+      "eval_steps_per_second": 0.175,
+      "step": 140
+    },
+    {
+      "epoch": 0.7955182072829131,
+      "eval_logits/chosen": -2.546875,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -434.0,
+      "eval_logps/rejected": -516.0,
+      "eval_loss": 0.40878039598464966,
+      "eval_rewards/accuracies": 0.875,
+      "eval_rewards/chosen": -0.9140625,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.34375,
+      "eval_runtime": 138.3436,
+      "eval_samples_per_second": 10.749,
+      "eval_steps_per_second": 0.173,
+      "step": 142
+    },
+    {
+      "epoch": 0.8067226890756303,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -430.0,
+      "eval_logps/rejected": -516.0,
+      "eval_loss": 0.40911930799484253,
+      "eval_rewards/accuracies": 0.875,
+      "eval_rewards/chosen": -0.89453125,
+      "eval_rewards/margins": 1.421875,
+      "eval_rewards/rejected": -2.3125,
+      "eval_runtime": 138.6927,
+      "eval_samples_per_second": 10.722,
+      "eval_steps_per_second": 0.173,
+      "step": 144
+    },
+    {
+      "epoch": 0.8179271708683473,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -430.0,
+      "eval_logps/rejected": -512.0,
+      "eval_loss": 0.40929388999938965,
+      "eval_rewards/accuracies": 0.8723958134651184,
+      "eval_rewards/chosen": -0.875,
+      "eval_rewards/margins": 1.40625,
+      "eval_rewards/rejected": -2.28125,
+      "eval_runtime": 137.8032,
+      "eval_samples_per_second": 10.791,
+      "eval_steps_per_second": 0.174,
+      "step": 146
+    },
+    {
+      "epoch": 0.8291316526610645,
+      "eval_logits/chosen": -2.546875,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -428.0,
+      "eval_logps/rejected": -510.0,
+      "eval_loss": 0.40948250889778137,
+      "eval_rewards/accuracies": 0.8776041865348816,
+      "eval_rewards/chosen": -0.8671875,
+      "eval_rewards/margins": 1.40625,
+      "eval_rewards/rejected": -2.265625,
+      "eval_runtime": 137.9295,
+      "eval_samples_per_second": 10.781,
+      "eval_steps_per_second": 0.174,
+      "step": 148
+    },
+    {
+      "epoch": 0.8403361344537815,
+      "grad_norm": 13.377221899659885,
+      "learning_rate": 3.683995891147695e-08,
+      "logits/chosen": -2.5625,
+      "logits/rejected": -2.46875,
+      "logps/chosen": -422.0,
+      "logps/rejected": -524.0,
+      "loss": 0.3992,
+      "rewards/accuracies": 0.8734375238418579,
+      "rewards/chosen": -0.8828125,
+      "rewards/margins": 1.4609375,
+      "rewards/rejected": -2.34375,
+      "step": 150
+    },
+    {
+      "epoch": 0.8403361344537815,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -430.0,
+      "eval_logps/rejected": -512.0,
+      "eval_loss": 0.4091300666332245,
+      "eval_rewards/accuracies": 0.8697916865348816,
+      "eval_rewards/chosen": -0.87109375,
+      "eval_rewards/margins": 1.40625,
+      "eval_rewards/rejected": -2.28125,
+      "eval_runtime": 139.5918,
+      "eval_samples_per_second": 10.652,
+      "eval_steps_per_second": 0.172,
+      "step": 150
+    },
+    {
+      "epoch": 0.8515406162464986,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -430.0,
+      "eval_logps/rejected": -512.0,
+      "eval_loss": 0.40880611538887024,
+      "eval_rewards/accuracies": 0.8723958134651184,
+      "eval_rewards/chosen": -0.88671875,
+      "eval_rewards/margins": 1.4140625,
+      "eval_rewards/rejected": -2.296875,
+      "eval_runtime": 138.1583,
+      "eval_samples_per_second": 10.763,
+      "eval_steps_per_second": 0.174,
+      "step": 152
+    },
+    {
+      "epoch": 0.8627450980392157,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.515625,
+      "eval_logps/chosen": -432.0,
+      "eval_logps/rejected": -516.0,
+      "eval_loss": 0.4081113338470459,
+      "eval_rewards/accuracies": 0.8697916865348816,
+      "eval_rewards/chosen": -0.91015625,
+      "eval_rewards/margins": 1.4140625,
+      "eval_rewards/rejected": -2.328125,
+      "eval_runtime": 138.4751,
+      "eval_samples_per_second": 10.738,
+      "eval_steps_per_second": 0.173,
+      "step": 154
+    },
+    {
+      "epoch": 0.8739495798319328,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -436.0,
+      "eval_logps/rejected": -520.0,
+      "eval_loss": 0.4078213572502136,
+      "eval_rewards/accuracies": 0.875,
+      "eval_rewards/chosen": -0.93359375,
+      "eval_rewards/margins": 1.421875,
+      "eval_rewards/rejected": -2.359375,
+      "eval_runtime": 137.9396,
+      "eval_samples_per_second": 10.78,
+      "eval_steps_per_second": 0.174,
+      "step": 156
+    },
+    {
+      "epoch": 0.8851540616246498,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -436.0,
+      "eval_logps/rejected": -520.0,
+      "eval_loss": 0.4075515568256378,
+      "eval_rewards/accuracies": 0.8723958134651184,
+      "eval_rewards/chosen": -0.94921875,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.375,
+      "eval_runtime": 137.7413,
+      "eval_samples_per_second": 10.796,
+      "eval_steps_per_second": 0.174,
+      "step": 158
+    },
+    {
+      "epoch": 0.896358543417367,
+      "grad_norm": 19.80329066156204,
+      "learning_rate": 1.5452166019378987e-08,
+      "logits/chosen": -2.59375,
+      "logits/rejected": -2.5,
+      "logps/chosen": -426.0,
+      "logps/rejected": -510.0,
+      "loss": 0.4087,
+      "rewards/accuracies": 0.878125011920929,
+      "rewards/chosen": -0.94921875,
+      "rewards/margins": 1.3828125,
+      "rewards/rejected": -2.328125,
+      "step": 160
+    },
+    {
+      "epoch": 0.896358543417367,
+      "eval_logits/chosen": -2.53125,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -438.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.40748268365859985,
+      "eval_rewards/accuracies": 0.8671875,
+      "eval_rewards/chosen": -0.96484375,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.390625,
+      "eval_runtime": 138.1961,
+      "eval_samples_per_second": 10.76,
+      "eval_steps_per_second": 0.174,
+      "step": 160
+    },
+    {
+      "epoch": 0.907563025210084,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -438.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.40728282928466797,
+      "eval_rewards/accuracies": 0.8697916865348816,
+      "eval_rewards/chosen": -0.96875,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.40625,
+      "eval_runtime": 138.4829,
+      "eval_samples_per_second": 10.738,
+      "eval_steps_per_second": 0.173,
+      "step": 162
+    },
+    {
+      "epoch": 0.9187675070028011,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -440.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.40710970759391785,
+      "eval_rewards/accuracies": 0.8645833134651184,
+      "eval_rewards/chosen": -0.97265625,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.40625,
+      "eval_runtime": 137.9278,
+      "eval_samples_per_second": 10.781,
+      "eval_steps_per_second": 0.174,
+      "step": 164
+    },
+    {
+      "epoch": 0.9299719887955182,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -440.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.4071543216705322,
+      "eval_rewards/accuracies": 0.8671875,
+      "eval_rewards/chosen": -0.9765625,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.40625,
+      "eval_runtime": 137.9153,
+      "eval_samples_per_second": 10.782,
+      "eval_steps_per_second": 0.174,
+      "step": 166
+    },
+    {
+      "epoch": 0.9411764705882353,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -440.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.4072038531303406,
+      "eval_rewards/accuracies": 0.8645833134651184,
+      "eval_rewards/chosen": -0.9765625,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.40625,
+      "eval_runtime": 139.0326,
+      "eval_samples_per_second": 10.695,
+      "eval_steps_per_second": 0.173,
+      "step": 168
+    },
+    {
+      "epoch": 0.9523809523809523,
+      "grad_norm": 12.662453014389987,
+      "learning_rate": 3.077914851215585e-09,
+      "logits/chosen": -2.53125,
+      "logits/rejected": -2.5,
+      "logps/chosen": -428.0,
+      "logps/rejected": -532.0,
+      "loss": 0.4016,
+      "rewards/accuracies": 0.8812500238418579,
+      "rewards/chosen": -1.0078125,
+      "rewards/margins": 1.5625,
+      "rewards/rejected": -2.578125,
+      "step": 170
+    },
+    {
+      "epoch": 0.9523809523809523,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -440.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.40719640254974365,
+      "eval_rewards/accuracies": 0.8671875,
+      "eval_rewards/chosen": -0.9765625,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.40625,
+      "eval_runtime": 139.2912,
+      "eval_samples_per_second": 10.675,
+      "eval_steps_per_second": 0.172,
+      "step": 170
+    },
+    {
+      "epoch": 0.9635854341736695,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -440.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.4072449505329132,
+      "eval_rewards/accuracies": 0.8645833134651184,
+      "eval_rewards/chosen": -0.9765625,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.40625,
+      "eval_runtime": 139.3473,
+      "eval_samples_per_second": 10.671,
+      "eval_steps_per_second": 0.172,
+      "step": 172
+    },
+    {
+      "epoch": 0.9747899159663865,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -440.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.40715768933296204,
+      "eval_rewards/accuracies": 0.8723958134651184,
+      "eval_rewards/chosen": -0.9765625,
+      "eval_rewards/margins": 1.4375,
+      "eval_rewards/rejected": -2.40625,
+      "eval_runtime": 137.5339,
+      "eval_samples_per_second": 10.812,
+      "eval_steps_per_second": 0.175,
+      "step": 174
+    },
+    {
+      "epoch": 0.9859943977591037,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -440.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.40728598833084106,
+      "eval_rewards/accuracies": 0.8697916865348816,
+      "eval_rewards/chosen": -0.9765625,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.40625,
+      "eval_runtime": 138.4016,
+      "eval_samples_per_second": 10.744,
+      "eval_steps_per_second": 0.173,
+      "step": 176
+    },
+    {
+      "epoch": 0.9971988795518207,
+      "eval_logits/chosen": -2.515625,
+      "eval_logits/rejected": -2.5,
+      "eval_logps/chosen": -440.0,
+      "eval_logps/rejected": -524.0,
+      "eval_loss": 0.4071241021156311,
+      "eval_rewards/accuracies": 0.8671875,
+      "eval_rewards/chosen": -0.9765625,
+      "eval_rewards/margins": 1.4296875,
+      "eval_rewards/rejected": -2.40625,
+      "eval_runtime": 139.1998,
+      "eval_samples_per_second": 10.682,
+      "eval_steps_per_second": 0.172,
+      "step": 178
+    },
+    {
+      "epoch": 0.9971988795518207,
+      "step": 178,
+      "total_flos": 0.0,
+      "train_loss": 0.47946612754564605,
+      "train_runtime": 42499.5655,
+      "train_samples_per_second": 1.073,
+      "train_steps_per_second": 0.004
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 178,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 2,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:79c197abab6cbb7c63234c86334d214f471bce9fcc36bdff48efe33d229a737a
+oid sha256:db18c50eb67e7d41695ef6c83dcbc7d49d84325facd8fb47cfc5eff848169de3
 size 7736