Upload folder using huggingface_hub

README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04bc4e5e61b25956286dfb171089a6a0a5f2b7296469a67df9f4a0554cfdf2b7
+size 25172088

checkpoint-3125/README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

checkpoint-3125/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-3125/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04bc4e5e61b25956286dfb171089a6a0a5f2b7296469a67df9f4a0554cfdf2b7
+size 25172088

checkpoint-3125/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cacb42485c55e1e15b1740506e21d6de8a046e87aa6ca9a48170d1c643fc2114
+size 50372538

checkpoint-3125/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5994634e90ce1b45d012a45568063be05fea876e791cd66b48a4efc924164b2
+size 14244

checkpoint-3125/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e42e57efa8b95889e87232fe6ea0805483ed6c8a2d35bb60230715b624ff5bda
+size 1064

checkpoint-3125/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-3125/tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-3125/trainer_state.json

@@ -0,0 +1,2217 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 3125,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0032,
+      "grad_norm": 5.0,
+      "learning_rate": 6.369426751592357e-06,
+      "loss": 22.4391,
+      "step": 10
+    },
+    {
+      "epoch": 0.0064,
+      "grad_norm": 5.0,
+      "learning_rate": 1.2738853503184714e-05,
+      "loss": 18.5398,
+      "step": 20
+    },
+    {
+      "epoch": 0.0096,
+      "grad_norm": 5.000000476837158,
+      "learning_rate": 1.910828025477707e-05,
+      "loss": 18.8691,
+      "step": 30
+    },
+    {
+      "epoch": 0.0128,
+      "grad_norm": 5.0,
+      "learning_rate": 2.5477707006369428e-05,
+      "loss": 17.9809,
+      "step": 40
+    },
+    {
+      "epoch": 0.016,
+      "grad_norm": 5.0,
+      "learning_rate": 3.184713375796178e-05,
+      "loss": 15.009,
+      "step": 50
+    },
+    {
+      "epoch": 0.0192,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 3.821656050955414e-05,
+      "loss": 14.248,
+      "step": 60
+    },
+    {
+      "epoch": 0.0224,
+      "grad_norm": 5.0,
+      "learning_rate": 4.45859872611465e-05,
+      "loss": 11.8696,
+      "step": 70
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 5.0955414012738855e-05,
+      "loss": 9.5315,
+      "step": 80
+    },
+    {
+      "epoch": 0.0288,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 5.732484076433121e-05,
+      "loss": 7.9867,
+      "step": 90
+    },
+    {
+      "epoch": 0.032,
+      "grad_norm": 5.0,
+      "learning_rate": 6.369426751592356e-05,
+      "loss": 6.6488,
+      "step": 100
+    },
+    {
+      "epoch": 0.0352,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.006369426751592e-05,
+      "loss": 4.6404,
+      "step": 110
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.643312101910829e-05,
+      "loss": 4.1932,
+      "step": 120
+    },
+    {
+      "epoch": 0.0416,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.280254777070065e-05,
+      "loss": 3.4752,
+      "step": 130
+    },
+    {
+      "epoch": 0.0448,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.9171974522293e-05,
+      "loss": 2.8687,
+      "step": 140
+    },
+    {
+      "epoch": 0.048,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.554140127388536e-05,
+      "loss": 2.5653,
+      "step": 150
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.989892183288411e-05,
+      "loss": 2.2596,
+      "step": 160
+    },
+    {
+      "epoch": 0.0544,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.956199460916442e-05,
+      "loss": 1.9221,
+      "step": 170
+    },
+    {
+      "epoch": 0.0576,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.922506738544474e-05,
+      "loss": 1.7448,
+      "step": 180
+    },
+    {
+      "epoch": 0.0608,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.888814016172507e-05,
+      "loss": 1.6752,
+      "step": 190
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 4.960172653198242,
+      "learning_rate": 9.85512129380054e-05,
+      "loss": 1.5561,
+      "step": 200
+    },
+    {
+      "epoch": 0.0672,
+      "grad_norm": 4.512001991271973,
+      "learning_rate": 9.821428571428572e-05,
+      "loss": 1.4908,
+      "step": 210
+    },
+    {
+      "epoch": 0.0704,
+      "grad_norm": 4.827399253845215,
+      "learning_rate": 9.787735849056603e-05,
+      "loss": 1.2773,
+      "step": 220
+    },
+    {
+      "epoch": 0.0736,
+      "grad_norm": 3.5412518978118896,
+      "learning_rate": 9.754043126684636e-05,
+      "loss": 1.2566,
+      "step": 230
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 4.866581439971924,
+      "learning_rate": 9.720350404312669e-05,
+      "loss": 1.3037,
+      "step": 240
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 3.3435661792755127,
+      "learning_rate": 9.686657681940702e-05,
+      "loss": 1.1427,
+      "step": 250
+    },
+    {
+      "epoch": 0.0832,
+      "grad_norm": 4.5573577880859375,
+      "learning_rate": 9.652964959568734e-05,
+      "loss": 1.19,
+      "step": 260
+    },
+    {
+      "epoch": 0.0864,
+      "grad_norm": 4.2881903648376465,
+      "learning_rate": 9.619272237196765e-05,
+      "loss": 1.2111,
+      "step": 270
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 3.5418100357055664,
+      "learning_rate": 9.585579514824798e-05,
+      "loss": 1.1723,
+      "step": 280
+    },
+    {
+      "epoch": 0.0928,
+      "grad_norm": 3.8908731937408447,
+      "learning_rate": 9.551886792452831e-05,
+      "loss": 1.0575,
+      "step": 290
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 3.236272096633911,
+      "learning_rate": 9.518194070080863e-05,
+      "loss": 1.0798,
+      "step": 300
+    },
+    {
+      "epoch": 0.0992,
+      "grad_norm": 4.228330135345459,
+      "learning_rate": 9.484501347708896e-05,
+      "loss": 1.0962,
+      "step": 310
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 4.129443645477295,
+      "learning_rate": 9.450808625336927e-05,
+      "loss": 1.0134,
+      "step": 320
+    },
+    {
+      "epoch": 0.1056,
+      "grad_norm": 4.085168361663818,
+      "learning_rate": 9.41711590296496e-05,
+      "loss": 0.9604,
+      "step": 330
+    },
+    {
+      "epoch": 0.1088,
+      "grad_norm": 3.8179402351379395,
+      "learning_rate": 9.383423180592993e-05,
+      "loss": 1.0164,
+      "step": 340
+    },
+    {
+      "epoch": 0.112,
+      "grad_norm": 3.624455451965332,
+      "learning_rate": 9.349730458221025e-05,
+      "loss": 0.9685,
+      "step": 350
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 3.5315566062927246,
+      "learning_rate": 9.316037735849057e-05,
+      "loss": 0.8864,
+      "step": 360
+    },
+    {
+      "epoch": 0.1184,
+      "grad_norm": 3.7452166080474854,
+      "learning_rate": 9.282345013477089e-05,
+      "loss": 0.9273,
+      "step": 370
+    },
+    {
+      "epoch": 0.1216,
+      "grad_norm": 4.054487228393555,
+      "learning_rate": 9.248652291105122e-05,
+      "loss": 0.8983,
+      "step": 380
+    },
+    {
+      "epoch": 0.1248,
+      "grad_norm": 4.036263942718506,
+      "learning_rate": 9.214959568733154e-05,
+      "loss": 0.8788,
+      "step": 390
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 3.101487874984741,
+      "learning_rate": 9.181266846361186e-05,
+      "loss": 0.7646,
+      "step": 400
+    },
+    {
+      "epoch": 0.1312,
+      "grad_norm": 3.928171157836914,
+      "learning_rate": 9.14757412398922e-05,
+      "loss": 0.8178,
+      "step": 410
+    },
+    {
+      "epoch": 0.1344,
+      "grad_norm": 4.1417059898376465,
+      "learning_rate": 9.113881401617251e-05,
+      "loss": 0.7828,
+      "step": 420
+    },
+    {
+      "epoch": 0.1376,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.080188679245284e-05,
+      "loss": 0.8381,
+      "step": 430
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 4.596191883087158,
+      "learning_rate": 9.046495956873315e-05,
+      "loss": 0.8254,
+      "step": 440
+    },
+    {
+      "epoch": 0.144,
+      "grad_norm": 3.0273566246032715,
+      "learning_rate": 9.012803234501348e-05,
+      "loss": 0.772,
+      "step": 450
+    },
+    {
+      "epoch": 0.1472,
+      "grad_norm": 2.827469825744629,
+      "learning_rate": 8.979110512129381e-05,
+      "loss": 0.7263,
+      "step": 460
+    },
+    {
+      "epoch": 0.1504,
+      "grad_norm": 3.2652060985565186,
+      "learning_rate": 8.945417789757413e-05,
+      "loss": 0.7221,
+      "step": 470
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 3.0766632556915283,
+      "learning_rate": 8.911725067385444e-05,
+      "loss": 0.702,
+      "step": 480
+    },
+    {
+      "epoch": 0.1568,
+      "grad_norm": 2.8645482063293457,
+      "learning_rate": 8.878032345013477e-05,
+      "loss": 0.6418,
+      "step": 490
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 3.1364777088165283,
+      "learning_rate": 8.84433962264151e-05,
+      "loss": 0.7149,
+      "step": 500
+    },
+    {
+      "epoch": 0.1632,
+      "grad_norm": 4.046690940856934,
+      "learning_rate": 8.810646900269543e-05,
+      "loss": 0.6353,
+      "step": 510
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 3.265176296234131,
+      "learning_rate": 8.776954177897575e-05,
+      "loss": 0.6268,
+      "step": 520
+    },
+    {
+      "epoch": 0.1696,
+      "grad_norm": 2.857509136199951,
+      "learning_rate": 8.743261455525606e-05,
+      "loss": 0.5919,
+      "step": 530
+    },
+    {
+      "epoch": 0.1728,
+      "grad_norm": 3.4214327335357666,
+      "learning_rate": 8.709568733153639e-05,
+      "loss": 0.6128,
+      "step": 540
+    },
+    {
+      "epoch": 0.176,
+      "grad_norm": 4.073673725128174,
+      "learning_rate": 8.675876010781672e-05,
+      "loss": 0.6164,
+      "step": 550
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 3.854487895965576,
+      "learning_rate": 8.642183288409704e-05,
+      "loss": 0.6125,
+      "step": 560
+    },
+    {
+      "epoch": 0.1824,
+      "grad_norm": 3.6991710662841797,
+      "learning_rate": 8.608490566037735e-05,
+      "loss": 0.5139,
+      "step": 570
+    },
+    {
+      "epoch": 0.1856,
+      "grad_norm": 3.3676042556762695,
+      "learning_rate": 8.574797843665768e-05,
+      "loss": 0.5112,
+      "step": 580
+    },
+    {
+      "epoch": 0.1888,
+      "grad_norm": 2.7846508026123047,
+      "learning_rate": 8.541105121293801e-05,
+      "loss": 0.5244,
+      "step": 590
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 3.6226212978363037,
+      "learning_rate": 8.507412398921834e-05,
+      "loss": 0.4886,
+      "step": 600
+    },
+    {
+      "epoch": 0.1952,
+      "grad_norm": 4.951099395751953,
+      "learning_rate": 8.473719676549866e-05,
+      "loss": 0.5436,
+      "step": 610
+    },
+    {
+      "epoch": 0.1984,
+      "grad_norm": 3.92474102973938,
+      "learning_rate": 8.440026954177897e-05,
+      "loss": 0.4834,
+      "step": 620
+    },
+    {
+      "epoch": 0.2016,
+      "grad_norm": 3.5895919799804688,
+      "learning_rate": 8.40633423180593e-05,
+      "loss": 0.4462,
+      "step": 630
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 3.756981372833252,
+      "learning_rate": 8.372641509433963e-05,
+      "loss": 0.4694,
+      "step": 640
+    },
+    {
+      "epoch": 0.208,
+      "grad_norm": 3.021611213684082,
+      "learning_rate": 8.338948787061996e-05,
+      "loss": 0.4597,
+      "step": 650
+    },
+    {
+      "epoch": 0.2112,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.305256064690027e-05,
+      "loss": 0.3989,
+      "step": 660
+    },
+    {
+      "epoch": 0.2144,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.271563342318059e-05,
+      "loss": 0.4079,
+      "step": 670
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 3.6756749153137207,
+      "learning_rate": 8.237870619946092e-05,
+      "loss": 0.3751,
+      "step": 680
+    },
+    {
+      "epoch": 0.2208,
+      "grad_norm": 2.9354946613311768,
+      "learning_rate": 8.204177897574125e-05,
+      "loss": 0.3304,
+      "step": 690
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.170485175202158e-05,
+      "loss": 0.3414,
+      "step": 700
+    },
+    {
+      "epoch": 0.2272,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.136792452830189e-05,
+      "loss": 0.3368,
+      "step": 710
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 2.68564772605896,
+      "learning_rate": 8.103099730458221e-05,
+      "loss": 0.2955,
+      "step": 720
+    },
+    {
+      "epoch": 0.2336,
+      "grad_norm": 3.137021780014038,
+      "learning_rate": 8.069407008086254e-05,
+      "loss": 0.2871,
+      "step": 730
+    },
+    {
+      "epoch": 0.2368,
+      "grad_norm": 2.4497194290161133,
+      "learning_rate": 8.035714285714287e-05,
+      "loss": 0.305,
+      "step": 740
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 3.228701114654541,
+      "learning_rate": 8.002021563342318e-05,
+      "loss": 0.2434,
+      "step": 750
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 2.0787534713745117,
+      "learning_rate": 7.968328840970351e-05,
+      "loss": 0.2763,
+      "step": 760
+    },
+    {
+      "epoch": 0.2464,
+      "grad_norm": 3.1535820960998535,
+      "learning_rate": 7.934636118598383e-05,
+      "loss": 0.2501,
+      "step": 770
+    },
+    {
+      "epoch": 0.2496,
+      "grad_norm": 1.965580701828003,
+      "learning_rate": 7.900943396226416e-05,
+      "loss": 0.2706,
+      "step": 780
+    },
+    {
+      "epoch": 0.2528,
+      "grad_norm": 2.9210808277130127,
+      "learning_rate": 7.867250673854449e-05,
+      "loss": 0.2359,
+      "step": 790
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 3.354069232940674,
+      "learning_rate": 7.83355795148248e-05,
+      "loss": 0.2389,
+      "step": 800
+    },
+    {
+      "epoch": 0.2592,
+      "grad_norm": 2.353992223739624,
+      "learning_rate": 7.799865229110512e-05,
+      "loss": 0.1947,
+      "step": 810
+    },
+    {
+      "epoch": 0.2624,
+      "grad_norm": 2.1019110679626465,
+      "learning_rate": 7.766172506738545e-05,
+      "loss": 0.196,
+      "step": 820
+    },
+    {
+      "epoch": 0.2656,
+      "grad_norm": 2.902089834213257,
+      "learning_rate": 7.732479784366577e-05,
+      "loss": 0.2167,
+      "step": 830
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 2.7410049438476562,
+      "learning_rate": 7.69878706199461e-05,
+      "loss": 0.1899,
+      "step": 840
+    },
+    {
+      "epoch": 0.272,
+      "grad_norm": 2.143695116043091,
+      "learning_rate": 7.665094339622642e-05,
+      "loss": 0.1795,
+      "step": 850
+    },
+    {
+      "epoch": 0.2752,
+      "grad_norm": 1.853039264678955,
+      "learning_rate": 7.631401617250674e-05,
+      "loss": 0.2081,
+      "step": 860
+    },
+    {
+      "epoch": 0.2784,
+      "grad_norm": 2.7920002937316895,
+      "learning_rate": 7.597708894878706e-05,
+      "loss": 0.2033,
+      "step": 870
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 1.7611720561981201,
+      "learning_rate": 7.56401617250674e-05,
+      "loss": 0.1762,
+      "step": 880
+    },
+    {
+      "epoch": 0.2848,
+      "grad_norm": 2.224700450897217,
+      "learning_rate": 7.530323450134771e-05,
+      "loss": 0.1721,
+      "step": 890
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 1.5112501382827759,
+      "learning_rate": 7.496630727762804e-05,
+      "loss": 0.166,
+      "step": 900
+    },
+    {
+      "epoch": 0.2912,
+      "grad_norm": 1.5669995546340942,
+      "learning_rate": 7.462938005390835e-05,
+      "loss": 0.1605,
+      "step": 910
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 1.366951584815979,
+      "learning_rate": 7.429245283018868e-05,
+      "loss": 0.1565,
+      "step": 920
+    },
+    {
+      "epoch": 0.2976,
+      "grad_norm": 3.0640170574188232,
+      "learning_rate": 7.395552560646901e-05,
+      "loss": 0.1562,
+      "step": 930
+    },
+    {
+      "epoch": 0.3008,
+      "grad_norm": 1.766140103340149,
+      "learning_rate": 7.361859838274933e-05,
+      "loss": 0.1568,
+      "step": 940
+    },
+    {
+      "epoch": 0.304,
+      "grad_norm": 1.9538131952285767,
+      "learning_rate": 7.328167115902966e-05,
+      "loss": 0.1554,
+      "step": 950
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 1.3904852867126465,
+      "learning_rate": 7.294474393530997e-05,
+      "loss": 0.1316,
+      "step": 960
+    },
+    {
+      "epoch": 0.3104,
+      "grad_norm": 1.8764123916625977,
+      "learning_rate": 7.26078167115903e-05,
+      "loss": 0.1653,
+      "step": 970
+    },
+    {
+      "epoch": 0.3136,
+      "grad_norm": 1.753258228302002,
+      "learning_rate": 7.227088948787062e-05,
+      "loss": 0.1299,
+      "step": 980
+    },
+    {
+      "epoch": 0.3168,
+      "grad_norm": 1.5121208429336548,
+      "learning_rate": 7.193396226415095e-05,
+      "loss": 0.1332,
+      "step": 990
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 1.6741907596588135,
+      "learning_rate": 7.159703504043128e-05,
+      "loss": 0.1375,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3232,
+      "grad_norm": 1.113685131072998,
+      "learning_rate": 7.126010781671159e-05,
+      "loss": 0.1345,
+      "step": 1010
+    },
+    {
+      "epoch": 0.3264,
+      "grad_norm": 2.015662431716919,
+      "learning_rate": 7.092318059299192e-05,
+      "loss": 0.1346,
+      "step": 1020
+    },
+    {
+      "epoch": 0.3296,
+      "grad_norm": 1.538110375404358,
+      "learning_rate": 7.058625336927224e-05,
+      "loss": 0.1297,
+      "step": 1030
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 1.3940174579620361,
+      "learning_rate": 7.024932614555257e-05,
+      "loss": 0.1331,
+      "step": 1040
+    },
+    {
+      "epoch": 0.336,
+      "grad_norm": 2.0724356174468994,
+      "learning_rate": 6.99123989218329e-05,
+      "loss": 0.1328,
+      "step": 1050
+    },
+    {
+      "epoch": 0.3392,
+      "grad_norm": 1.7255312204360962,
+      "learning_rate": 6.957547169811321e-05,
+      "loss": 0.1322,
+      "step": 1060
+    },
+    {
+      "epoch": 0.3424,
+      "grad_norm": 2.1497256755828857,
+      "learning_rate": 6.923854447439353e-05,
+      "loss": 0.1415,
+      "step": 1070
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 1.7581284046173096,
+      "learning_rate": 6.890161725067386e-05,
+      "loss": 0.1227,
+      "step": 1080
+    },
+    {
+      "epoch": 0.3488,
+      "grad_norm": 1.199498176574707,
+      "learning_rate": 6.856469002695418e-05,
+      "loss": 0.1285,
+      "step": 1090
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 1.792433500289917,
+      "learning_rate": 6.822776280323451e-05,
+      "loss": 0.1204,
+      "step": 1100
+    },
+    {
+      "epoch": 0.3552,
+      "grad_norm": 1.1998846530914307,
+      "learning_rate": 6.789083557951483e-05,
+      "loss": 0.1126,
+      "step": 1110
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 1.4056320190429688,
+      "learning_rate": 6.755390835579514e-05,
+      "loss": 0.1137,
+      "step": 1120
+    },
+    {
+      "epoch": 0.3616,
+      "grad_norm": 2.3784384727478027,
+      "learning_rate": 6.721698113207547e-05,
+      "loss": 0.1156,
+      "step": 1130
+    },
+    {
+      "epoch": 0.3648,
+      "grad_norm": 1.422841191291809,
+      "learning_rate": 6.68800539083558e-05,
+      "loss": 0.1179,
+      "step": 1140
+    },
+    {
+      "epoch": 0.368,
+      "grad_norm": 1.3304040431976318,
+      "learning_rate": 6.654312668463612e-05,
+      "loss": 0.1135,
+      "step": 1150
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 1.1918659210205078,
+      "learning_rate": 6.620619946091643e-05,
+      "loss": 0.1146,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3744,
+      "grad_norm": 1.455103874206543,
+      "learning_rate": 6.586927223719676e-05,
+      "loss": 0.1124,
+      "step": 1170
+    },
+    {
+      "epoch": 0.3776,
+      "grad_norm": 1.1681839227676392,
+      "learning_rate": 6.553234501347709e-05,
+      "loss": 0.1016,
+      "step": 1180
+    },
+    {
+      "epoch": 0.3808,
+      "grad_norm": 2.063532829284668,
+      "learning_rate": 6.519541778975742e-05,
+      "loss": 0.1067,
+      "step": 1190
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 2.1005892753601074,
+      "learning_rate": 6.485849056603774e-05,
+      "loss": 0.103,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3872,
+      "grad_norm": 1.8993762731552124,
+      "learning_rate": 6.452156334231805e-05,
+      "loss": 0.1016,
+      "step": 1210
+    },
+    {
+      "epoch": 0.3904,
+      "grad_norm": 1.246466875076294,
+      "learning_rate": 6.418463611859838e-05,
+      "loss": 0.1139,
+      "step": 1220
+    },
+    {
+      "epoch": 0.3936,
+      "grad_norm": 1.0742267370224,
+      "learning_rate": 6.384770889487871e-05,
+      "loss": 0.0944,
+      "step": 1230
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 1.6990922689437866,
+      "learning_rate": 6.351078167115904e-05,
+      "loss": 0.1047,
+      "step": 1240
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.9877095818519592,
+      "learning_rate": 6.317385444743936e-05,
+      "loss": 0.0984,
+      "step": 1250
+    },
+    {
+      "epoch": 0.4032,
+      "grad_norm": 2.03071665763855,
+      "learning_rate": 6.283692722371967e-05,
+      "loss": 0.1018,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4064,
+      "grad_norm": 1.1828073263168335,
+      "learning_rate": 6.25e-05,
+      "loss": 0.099,
+      "step": 1270
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 2.286886692047119,
+      "learning_rate": 6.216307277628033e-05,
+      "loss": 0.1119,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4128,
+      "grad_norm": 1.35301673412323,
+      "learning_rate": 6.182614555256066e-05,
+      "loss": 0.1062,
+      "step": 1290
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 1.1716175079345703,
+      "learning_rate": 6.148921832884098e-05,
+      "loss": 0.0919,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4192,
+      "grad_norm": 1.2347395420074463,
+      "learning_rate": 6.115229110512129e-05,
+      "loss": 0.0941,
+      "step": 1310
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 1.1255364418029785,
+      "learning_rate": 6.081536388140162e-05,
+      "loss": 0.0917,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4256,
+      "grad_norm": 1.3256624937057495,
+      "learning_rate": 6.047843665768195e-05,
+      "loss": 0.0945,
+      "step": 1330
+    },
+    {
+      "epoch": 0.4288,
+      "grad_norm": 1.4447101354599,
+      "learning_rate": 6.0141509433962265e-05,
+      "loss": 0.104,
+      "step": 1340
+    },
+    {
+      "epoch": 0.432,
+      "grad_norm": 1.0093165636062622,
+      "learning_rate": 5.980458221024259e-05,
+      "loss": 0.0856,
+      "step": 1350
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 2.2328953742980957,
+      "learning_rate": 5.9467654986522916e-05,
+      "loss": 0.0957,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4384,
+      "grad_norm": 1.4997823238372803,
+      "learning_rate": 5.913072776280324e-05,
+      "loss": 0.0933,
+      "step": 1370
+    },
+    {
+      "epoch": 0.4416,
+      "grad_norm": 0.8270460367202759,
+      "learning_rate": 5.879380053908357e-05,
+      "loss": 0.0961,
+      "step": 1380
+    },
+    {
+      "epoch": 0.4448,
+      "grad_norm": 1.8531345129013062,
+      "learning_rate": 5.8456873315363884e-05,
+      "loss": 0.085,
+      "step": 1390
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 1.0742558240890503,
+      "learning_rate": 5.8119946091644206e-05,
+      "loss": 0.0966,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4512,
+      "grad_norm": 1.1224265098571777,
+      "learning_rate": 5.7783018867924535e-05,
+      "loss": 0.0848,
+      "step": 1410
+    },
+    {
+      "epoch": 0.4544,
+      "grad_norm": 1.5263229608535767,
+      "learning_rate": 5.744609164420486e-05,
+      "loss": 0.0894,
+      "step": 1420
+    },
+    {
+      "epoch": 0.4576,
+      "grad_norm": 1.7091747522354126,
+      "learning_rate": 5.710916442048517e-05,
+      "loss": 0.0888,
+      "step": 1430
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 1.8716164827346802,
+      "learning_rate": 5.6772237196765496e-05,
+      "loss": 0.0843,
+      "step": 1440
+    },
+    {
+      "epoch": 0.464,
+      "grad_norm": 0.9060105681419373,
+      "learning_rate": 5.6435309973045825e-05,
+      "loss": 0.0869,
+      "step": 1450
+    },
+    {
+      "epoch": 0.4672,
+      "grad_norm": 1.1350743770599365,
+      "learning_rate": 5.609838274932615e-05,
+      "loss": 0.079,
+      "step": 1460
+    },
+    {
+      "epoch": 0.4704,
+      "grad_norm": 1.1754982471466064,
+      "learning_rate": 5.5761455525606476e-05,
+      "loss": 0.0835,
+      "step": 1470
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 1.6735552549362183,
+      "learning_rate": 5.542452830188679e-05,
+      "loss": 0.0884,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4768,
+      "grad_norm": 1.191200613975525,
+      "learning_rate": 5.5087601078167114e-05,
+      "loss": 0.0756,
+      "step": 1490
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 1.3234206438064575,
+      "learning_rate": 5.4750673854447444e-05,
+      "loss": 0.0685,
+      "step": 1500
+    },
+    {
+      "epoch": 0.4832,
+      "grad_norm": 1.2499926090240479,
+      "learning_rate": 5.4413746630727766e-05,
+      "loss": 0.0798,
+      "step": 1510
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 1.8494608402252197,
+      "learning_rate": 5.407681940700808e-05,
+      "loss": 0.0825,
+      "step": 1520
+    },
+    {
+      "epoch": 0.4896,
+      "grad_norm": 1.1932600736618042,
+      "learning_rate": 5.373989218328841e-05,
+      "loss": 0.0797,
+      "step": 1530
+    },
+    {
+      "epoch": 0.4928,
+      "grad_norm": 0.9455847144126892,
+      "learning_rate": 5.340296495956873e-05,
+      "loss": 0.074,
+      "step": 1540
+    },
+    {
+      "epoch": 0.496,
+      "grad_norm": 0.9006221294403076,
+      "learning_rate": 5.306603773584906e-05,
+      "loss": 0.077,
+      "step": 1550
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 1.0043009519577026,
+      "learning_rate": 5.2729110512129385e-05,
+      "loss": 0.0802,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5024,
+      "grad_norm": 1.027671456336975,
+      "learning_rate": 5.23921832884097e-05,
+      "loss": 0.0793,
+      "step": 1570
+    },
+    {
+      "epoch": 0.5056,
+      "grad_norm": 1.51658296585083,
+      "learning_rate": 5.205525606469003e-05,
+      "loss": 0.0793,
+      "step": 1580
+    },
+    {
+      "epoch": 0.5088,
+      "grad_norm": 1.336788535118103,
+      "learning_rate": 5.171832884097035e-05,
+      "loss": 0.0784,
+      "step": 1590
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 1.3772467374801636,
+      "learning_rate": 5.138140161725068e-05,
+      "loss": 0.0706,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5152,
+      "grad_norm": 0.922598123550415,
+      "learning_rate": 5.1044474393531e-05,
+      "loss": 0.0781,
+      "step": 1610
+    },
+    {
+      "epoch": 0.5184,
+      "grad_norm": 1.441529631614685,
+      "learning_rate": 5.070754716981132e-05,
+      "loss": 0.0825,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5216,
+      "grad_norm": 0.9344118237495422,
+      "learning_rate": 5.037061994609165e-05,
+      "loss": 0.0767,
+      "step": 1630
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 2.2413012981414795,
+      "learning_rate": 5.003369272237197e-05,
+      "loss": 0.0715,
+      "step": 1640
+    },
+    {
+      "epoch": 0.528,
+      "grad_norm": 1.0326889753341675,
+      "learning_rate": 4.969676549865229e-05,
+      "loss": 0.0814,
+      "step": 1650
+    },
+    {
+      "epoch": 0.5312,
+      "grad_norm": 1.0714298486709595,
+      "learning_rate": 4.9359838274932616e-05,
+      "loss": 0.0707,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5344,
+      "grad_norm": 1.345158338546753,
+      "learning_rate": 4.902291105121294e-05,
+      "loss": 0.0802,
+      "step": 1670
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 0.9195489883422852,
+      "learning_rate": 4.868598382749327e-05,
+      "loss": 0.0734,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5408,
+      "grad_norm": 0.7901739478111267,
+      "learning_rate": 4.834905660377358e-05,
+      "loss": 0.0744,
+      "step": 1690
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 1.0859198570251465,
+      "learning_rate": 4.801212938005391e-05,
+      "loss": 0.068,
+      "step": 1700
+    },
+    {
+      "epoch": 0.5472,
+      "grad_norm": 1.1353799104690552,
+      "learning_rate": 4.7675202156334234e-05,
+      "loss": 0.068,
+      "step": 1710
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 0.8822022080421448,
+      "learning_rate": 4.733827493261456e-05,
+      "loss": 0.0575,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5536,
+      "grad_norm": 0.9976137280464172,
+      "learning_rate": 4.7001347708894886e-05,
+      "loss": 0.0702,
+      "step": 1730
+    },
+    {
+      "epoch": 0.5568,
+      "grad_norm": 2.010540008544922,
+      "learning_rate": 4.66644204851752e-05,
+      "loss": 0.0715,
+      "step": 1740
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 2.2936131954193115,
+      "learning_rate": 4.632749326145553e-05,
+      "loss": 0.0714,
+      "step": 1750
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 0.8874325752258301,
+      "learning_rate": 4.5990566037735846e-05,
+      "loss": 0.0718,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5664,
+      "grad_norm": 0.8641786575317383,
+      "learning_rate": 4.5653638814016176e-05,
+      "loss": 0.068,
+      "step": 1770
+    },
+    {
+      "epoch": 0.5696,
+      "grad_norm": 1.0465822219848633,
+      "learning_rate": 4.53167115902965e-05,
+      "loss": 0.0772,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5728,
+      "grad_norm": 1.080430269241333,
+      "learning_rate": 4.497978436657682e-05,
+      "loss": 0.0703,
+      "step": 1790
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 1.214148759841919,
+      "learning_rate": 4.464285714285715e-05,
+      "loss": 0.0801,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5792,
+      "grad_norm": 1.1502673625946045,
+      "learning_rate": 4.4305929919137465e-05,
+      "loss": 0.0634,
+      "step": 1810
+    },
+    {
+      "epoch": 0.5824,
+      "grad_norm": 0.7351061105728149,
+      "learning_rate": 4.3969002695417794e-05,
+      "loss": 0.0592,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5856,
+      "grad_norm": 0.5843839049339294,
+      "learning_rate": 4.363207547169812e-05,
+      "loss": 0.0654,
+      "step": 1830
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 0.8455338478088379,
+      "learning_rate": 4.329514824797844e-05,
+      "loss": 0.0622,
+      "step": 1840
+    },
+    {
+      "epoch": 0.592,
+      "grad_norm": 1.4609434604644775,
+      "learning_rate": 4.295822102425876e-05,
+      "loss": 0.0665,
+      "step": 1850
+    },
+    {
+      "epoch": 0.5952,
+      "grad_norm": 0.9410058259963989,
+      "learning_rate": 4.2621293800539084e-05,
+      "loss": 0.0614,
+      "step": 1860
+    },
+    {
+      "epoch": 0.5984,
+      "grad_norm": 0.9071024060249329,
+      "learning_rate": 4.2284366576819406e-05,
+      "loss": 0.0622,
+      "step": 1870
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 1.175545334815979,
+      "learning_rate": 4.1947439353099736e-05,
+      "loss": 0.0645,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6048,
+      "grad_norm": 1.1436173915863037,
+      "learning_rate": 4.161051212938006e-05,
+      "loss": 0.0618,
+      "step": 1890
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 1.0221768617630005,
+      "learning_rate": 4.127358490566038e-05,
+      "loss": 0.0625,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6112,
+      "grad_norm": 1.6312997341156006,
+      "learning_rate": 4.09366576819407e-05,
+      "loss": 0.0649,
+      "step": 1910
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 0.963321328163147,
+      "learning_rate": 4.0599730458221025e-05,
+      "loss": 0.0689,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6176,
+      "grad_norm": 1.2729462385177612,
+      "learning_rate": 4.026280323450135e-05,
+      "loss": 0.0593,
+      "step": 1930
+    },
+    {
+      "epoch": 0.6208,
+      "grad_norm": 1.2127982378005981,
+      "learning_rate": 3.992587601078167e-05,
+      "loss": 0.0542,
+      "step": 1940
+    },
+    {
+      "epoch": 0.624,
+      "grad_norm": 0.7582926750183105,
+      "learning_rate": 3.9588948787062e-05,
+      "loss": 0.0626,
+      "step": 1950
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 1.59578275680542,
+      "learning_rate": 3.9252021563342315e-05,
+      "loss": 0.0629,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6304,
+      "grad_norm": 1.157958745956421,
+      "learning_rate": 3.8915094339622644e-05,
+      "loss": 0.0595,
+      "step": 1970
+    },
+    {
+      "epoch": 0.6336,
+      "grad_norm": 2.2247676849365234,
+      "learning_rate": 3.8578167115902966e-05,
+      "loss": 0.0601,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6368,
+      "grad_norm": 0.6778025031089783,
+      "learning_rate": 3.824123989218329e-05,
+      "loss": 0.0552,
+      "step": 1990
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 1.1828677654266357,
+      "learning_rate": 3.790431266846362e-05,
+      "loss": 0.0574,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6432,
+      "grad_norm": 0.7850990295410156,
+      "learning_rate": 3.7567385444743934e-05,
+      "loss": 0.0535,
+      "step": 2010
+    },
+    {
+      "epoch": 0.6464,
+      "grad_norm": 1.0724990367889404,
+      "learning_rate": 3.723045822102426e-05,
+      "loss": 0.0572,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6496,
+      "grad_norm": 1.0207794904708862,
+      "learning_rate": 3.689353099730458e-05,
+      "loss": 0.0571,
+      "step": 2030
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 0.7693538665771484,
+      "learning_rate": 3.655660377358491e-05,
+      "loss": 0.0626,
+      "step": 2040
+    },
+    {
+      "epoch": 0.656,
+      "grad_norm": 0.7968879342079163,
+      "learning_rate": 3.621967654986524e-05,
+      "loss": 0.0566,
+      "step": 2050
+    },
+    {
+      "epoch": 0.6592,
+      "grad_norm": 1.2296173572540283,
+      "learning_rate": 3.588274932614555e-05,
+      "loss": 0.0594,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6624,
+      "grad_norm": 0.8568419218063354,
+      "learning_rate": 3.554582210242588e-05,
+      "loss": 0.0513,
+      "step": 2070
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 0.9537982940673828,
+      "learning_rate": 3.52088948787062e-05,
+      "loss": 0.0547,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6688,
+      "grad_norm": 0.8022981882095337,
+      "learning_rate": 3.4871967654986526e-05,
+      "loss": 0.055,
+      "step": 2090
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 1.1287271976470947,
+      "learning_rate": 3.453504043126685e-05,
+      "loss": 0.0575,
+      "step": 2100
+    },
+    {
+      "epoch": 0.6752,
+      "grad_norm": 1.0856471061706543,
+      "learning_rate": 3.419811320754717e-05,
+      "loss": 0.0687,
+      "step": 2110
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 0.9026399850845337,
+      "learning_rate": 3.3861185983827494e-05,
+      "loss": 0.0523,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6816,
+      "grad_norm": 0.8722916841506958,
+      "learning_rate": 3.3524258760107816e-05,
+      "loss": 0.0545,
+      "step": 2130
+    },
+    {
+      "epoch": 0.6848,
+      "grad_norm": 1.5549736022949219,
+      "learning_rate": 3.3187331536388145e-05,
+      "loss": 0.054,
+      "step": 2140
+    },
+    {
+      "epoch": 0.688,
+      "grad_norm": 0.989742636680603,
+      "learning_rate": 3.285040431266847e-05,
+      "loss": 0.0536,
+      "step": 2150
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 1.279964566230774,
+      "learning_rate": 3.251347708894879e-05,
+      "loss": 0.0492,
+      "step": 2160
+    },
+    {
+      "epoch": 0.6944,
+      "grad_norm": 0.8057636022567749,
+      "learning_rate": 3.217654986522911e-05,
+      "loss": 0.0598,
+      "step": 2170
+    },
+    {
+      "epoch": 0.6976,
+      "grad_norm": 1.6911473274230957,
+      "learning_rate": 3.1839622641509435e-05,
+      "loss": 0.0626,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7008,
+      "grad_norm": 0.7519389986991882,
+      "learning_rate": 3.150269541778976e-05,
+      "loss": 0.0568,
+      "step": 2190
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 1.1423537731170654,
+      "learning_rate": 3.1165768194070086e-05,
+      "loss": 0.0584,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7072,
+      "grad_norm": 1.7910997867584229,
+      "learning_rate": 3.08288409703504e-05,
+      "loss": 0.0639,
+      "step": 2210
+    },
+    {
+      "epoch": 0.7104,
+      "grad_norm": 0.8631108403205872,
+      "learning_rate": 3.0491913746630728e-05,
+      "loss": 0.0577,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7136,
+      "grad_norm": 0.8344480395317078,
+      "learning_rate": 3.0154986522911054e-05,
+      "loss": 0.0526,
+      "step": 2230
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 1.616515874862671,
+      "learning_rate": 2.9818059299191376e-05,
+      "loss": 0.0564,
+      "step": 2240
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 1.0607733726501465,
+      "learning_rate": 2.9481132075471702e-05,
+      "loss": 0.0551,
+      "step": 2250
+    },
+    {
+      "epoch": 0.7232,
+      "grad_norm": 0.6787971258163452,
+      "learning_rate": 2.914420485175202e-05,
+      "loss": 0.0473,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7264,
+      "grad_norm": 1.0428708791732788,
+      "learning_rate": 2.8807277628032347e-05,
+      "loss": 0.0569,
+      "step": 2270
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 0.9818703532218933,
+      "learning_rate": 2.847035040431267e-05,
+      "loss": 0.0533,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7328,
+      "grad_norm": 1.5224767923355103,
+      "learning_rate": 2.8133423180592995e-05,
+      "loss": 0.0553,
+      "step": 2290
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 1.2571505308151245,
+      "learning_rate": 2.7796495956873314e-05,
+      "loss": 0.0532,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7392,
+      "grad_norm": 0.7414546608924866,
+      "learning_rate": 2.745956873315364e-05,
+      "loss": 0.0487,
+      "step": 2310
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 1.0934925079345703,
+      "learning_rate": 2.7122641509433965e-05,
+      "loss": 0.0557,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7456,
+      "grad_norm": 1.0424625873565674,
+      "learning_rate": 2.6785714285714288e-05,
+      "loss": 0.0494,
+      "step": 2330
+    },
+    {
+      "epoch": 0.7488,
+      "grad_norm": 1.1907474994659424,
+      "learning_rate": 2.6448787061994614e-05,
+      "loss": 0.0577,
+      "step": 2340
+    },
+    {
+      "epoch": 0.752,
+      "grad_norm": 1.0648528337478638,
+      "learning_rate": 2.6111859838274933e-05,
+      "loss": 0.0538,
+      "step": 2350
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 0.5466381907463074,
+      "learning_rate": 2.577493261455526e-05,
+      "loss": 0.0506,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7584,
+      "grad_norm": 1.2638976573944092,
+      "learning_rate": 2.5438005390835577e-05,
+      "loss": 0.0499,
+      "step": 2370
+    },
+    {
+      "epoch": 0.7616,
+      "grad_norm": 1.2526659965515137,
+      "learning_rate": 2.5101078167115903e-05,
+      "loss": 0.0518,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7648,
+      "grad_norm": 0.7721813917160034,
+      "learning_rate": 2.476415094339623e-05,
+      "loss": 0.049,
+      "step": 2390
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 0.6863059997558594,
+      "learning_rate": 2.442722371967655e-05,
+      "loss": 0.0526,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7712,
+      "grad_norm": 0.7994042634963989,
+      "learning_rate": 2.4090296495956874e-05,
+      "loss": 0.0499,
+      "step": 2410
+    },
+    {
+      "epoch": 0.7744,
+      "grad_norm": 1.0728378295898438,
+      "learning_rate": 2.3753369272237196e-05,
+      "loss": 0.0492,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7776,
+      "grad_norm": 1.2912821769714355,
+      "learning_rate": 2.341644204851752e-05,
+      "loss": 0.0461,
+      "step": 2430
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 0.9025666117668152,
+      "learning_rate": 2.3079514824797844e-05,
+      "loss": 0.0453,
+      "step": 2440
+    },
+    {
+      "epoch": 0.784,
+      "grad_norm": 0.6996231079101562,
+      "learning_rate": 2.274258760107817e-05,
+      "loss": 0.0471,
+      "step": 2450
+    },
+    {
+      "epoch": 0.7872,
+      "grad_norm": 0.7211071252822876,
+      "learning_rate": 2.2405660377358493e-05,
+      "loss": 0.0538,
+      "step": 2460
+    },
+    {
+      "epoch": 0.7904,
+      "grad_norm": 0.6897232532501221,
+      "learning_rate": 2.2068733153638815e-05,
+      "loss": 0.0484,
+      "step": 2470
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 1.3392776250839233,
+      "learning_rate": 2.1731805929919137e-05,
+      "loss": 0.0446,
+      "step": 2480
+    },
+    {
+      "epoch": 0.7968,
+      "grad_norm": 0.5439741015434265,
+      "learning_rate": 2.1394878706199463e-05,
+      "loss": 0.0467,
+      "step": 2490
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.6194649934768677,
+      "learning_rate": 2.1057951482479785e-05,
+      "loss": 0.0469,
+      "step": 2500
+    },
+    {
+      "epoch": 0.8032,
+      "grad_norm": 0.7839043736457825,
+      "learning_rate": 2.0721024258760108e-05,
+      "loss": 0.0459,
+      "step": 2510
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 0.9303894639015198,
+      "learning_rate": 2.038409703504043e-05,
+      "loss": 0.0483,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8096,
+      "grad_norm": 1.0741106271743774,
+      "learning_rate": 2.0047169811320756e-05,
+      "loss": 0.0464,
+      "step": 2530
+    },
+    {
+      "epoch": 0.8128,
+      "grad_norm": 0.9432321190834045,
+      "learning_rate": 1.971024258760108e-05,
+      "loss": 0.0477,
+      "step": 2540
+    },
+    {
+      "epoch": 0.816,
+      "grad_norm": 0.8696873188018799,
+      "learning_rate": 1.9373315363881404e-05,
+      "loss": 0.0504,
+      "step": 2550
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 1.1203161478042603,
+      "learning_rate": 1.9036388140161727e-05,
+      "loss": 0.0478,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8224,
+      "grad_norm": 1.0000349283218384,
+      "learning_rate": 1.869946091644205e-05,
+      "loss": 0.0518,
+      "step": 2570
+    },
+    {
+      "epoch": 0.8256,
+      "grad_norm": 0.7597396373748779,
+      "learning_rate": 1.836253369272237e-05,
+      "loss": 0.0487,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8288,
+      "grad_norm": 0.6735145449638367,
+      "learning_rate": 1.8025606469002694e-05,
+      "loss": 0.0458,
+      "step": 2590
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 0.7138289213180542,
+      "learning_rate": 1.768867924528302e-05,
+      "loss": 0.0408,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8352,
+      "grad_norm": 0.944825291633606,
+      "learning_rate": 1.7351752021563345e-05,
+      "loss": 0.0448,
+      "step": 2610
+    },
+    {
+      "epoch": 0.8384,
+      "grad_norm": 0.8002263307571411,
+      "learning_rate": 1.7014824797843668e-05,
+      "loss": 0.0465,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8416,
+      "grad_norm": 1.3529142141342163,
+      "learning_rate": 1.667789757412399e-05,
+      "loss": 0.0486,
+      "step": 2630
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 1.245259404182434,
+      "learning_rate": 1.6340970350404313e-05,
+      "loss": 0.05,
+      "step": 2640
+    },
+    {
+      "epoch": 0.848,
+      "grad_norm": 0.767680287361145,
+      "learning_rate": 1.600404312668464e-05,
+      "loss": 0.0493,
+      "step": 2650
+    },
+    {
+      "epoch": 0.8512,
+      "grad_norm": 1.5667990446090698,
+      "learning_rate": 1.566711590296496e-05,
+      "loss": 0.051,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8544,
+      "grad_norm": 1.1249333620071411,
+      "learning_rate": 1.5330188679245283e-05,
+      "loss": 0.0503,
+      "step": 2670
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 0.6794225573539734,
+      "learning_rate": 1.4993261455525606e-05,
+      "loss": 0.0404,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8608,
+      "grad_norm": 0.94537752866745,
+      "learning_rate": 1.465633423180593e-05,
+      "loss": 0.0466,
+      "step": 2690
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 0.9569212794303894,
+      "learning_rate": 1.4319407008086256e-05,
+      "loss": 0.0457,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8672,
+      "grad_norm": 0.7343612313270569,
+      "learning_rate": 1.3982479784366578e-05,
+      "loss": 0.0537,
+      "step": 2710
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 1.2045423984527588,
+      "learning_rate": 1.3645552560646902e-05,
+      "loss": 0.0447,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8736,
+      "grad_norm": 0.7130928635597229,
+      "learning_rate": 1.3308625336927224e-05,
+      "loss": 0.0477,
+      "step": 2730
+    },
+    {
+      "epoch": 0.8768,
+      "grad_norm": 1.2449103593826294,
+      "learning_rate": 1.2971698113207547e-05,
+      "loss": 0.048,
+      "step": 2740
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 1.1206499338150024,
+      "learning_rate": 1.2634770889487871e-05,
+      "loss": 0.044,
+      "step": 2750
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 0.7373448610305786,
+      "learning_rate": 1.2297843665768195e-05,
+      "loss": 0.0611,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8864,
+      "grad_norm": 1.2338485717773438,
+      "learning_rate": 1.1960916442048519e-05,
+      "loss": 0.0478,
+      "step": 2770
+    },
+    {
+      "epoch": 0.8896,
+      "grad_norm": 0.931049108505249,
+      "learning_rate": 1.1623989218328842e-05,
+      "loss": 0.0415,
+      "step": 2780
+    },
+    {
+      "epoch": 0.8928,
+      "grad_norm": 0.818911612033844,
+      "learning_rate": 1.1287061994609164e-05,
+      "loss": 0.0441,
+      "step": 2790
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 0.5560767650604248,
+      "learning_rate": 1.0950134770889488e-05,
+      "loss": 0.0445,
+      "step": 2800
+    },
+    {
+      "epoch": 0.8992,
+      "grad_norm": 0.6225351095199585,
+      "learning_rate": 1.0613207547169812e-05,
+      "loss": 0.0406,
+      "step": 2810
+    },
+    {
+      "epoch": 0.9024,
+      "grad_norm": 1.02946138381958,
+      "learning_rate": 1.0276280323450135e-05,
+      "loss": 0.0464,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9056,
+      "grad_norm": 0.7534985542297363,
+      "learning_rate": 9.939353099730459e-06,
+      "loss": 0.0508,
+      "step": 2830
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 0.7269020080566406,
+      "learning_rate": 9.602425876010781e-06,
+      "loss": 0.0479,
+      "step": 2840
+    },
+    {
+      "epoch": 0.912,
+      "grad_norm": 0.6104754209518433,
+      "learning_rate": 9.265498652291107e-06,
+      "loss": 0.0487,
+      "step": 2850
+    },
+    {
+      "epoch": 0.9152,
+      "grad_norm": 0.7579702734947205,
+      "learning_rate": 8.92857142857143e-06,
+      "loss": 0.0434,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9184,
+      "grad_norm": 0.7916242480278015,
+      "learning_rate": 8.591644204851752e-06,
+      "loss": 0.0447,
+      "step": 2870
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 0.9057757258415222,
+      "learning_rate": 8.254716981132076e-06,
+      "loss": 0.0468,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9248,
+      "grad_norm": 0.6375102400779724,
+      "learning_rate": 7.9177897574124e-06,
+      "loss": 0.0488,
+      "step": 2890
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 1.7158457040786743,
+      "learning_rate": 7.580862533692723e-06,
+      "loss": 0.0554,
+      "step": 2900
+    },
+    {
+      "epoch": 0.9312,
+      "grad_norm": 0.8057986497879028,
+      "learning_rate": 7.243935309973046e-06,
+      "loss": 0.0453,
+      "step": 2910
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 0.8072325587272644,
+      "learning_rate": 6.90700808625337e-06,
+      "loss": 0.0416,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9376,
+      "grad_norm": 0.6157782673835754,
+      "learning_rate": 6.570080862533692e-06,
+      "loss": 0.046,
+      "step": 2930
+    },
+    {
+      "epoch": 0.9408,
+      "grad_norm": 0.5398648977279663,
+      "learning_rate": 6.233153638814016e-06,
+      "loss": 0.0491,
+      "step": 2940
+    },
+    {
+      "epoch": 0.944,
+      "grad_norm": 1.032221794128418,
+      "learning_rate": 5.89622641509434e-06,
+      "loss": 0.0469,
+      "step": 2950
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 0.8197404742240906,
+      "learning_rate": 5.5592991913746634e-06,
+      "loss": 0.0441,
+      "step": 2960
+    },
+    {
+      "epoch": 0.9504,
+      "grad_norm": 0.7197135090827942,
+      "learning_rate": 5.222371967654987e-06,
+      "loss": 0.044,
+      "step": 2970
+    },
+    {
+      "epoch": 0.9536,
+      "grad_norm": 0.8980445861816406,
+      "learning_rate": 4.88544474393531e-06,
+      "loss": 0.046,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9568,
+      "grad_norm": 0.7673968076705933,
+      "learning_rate": 4.548517520215634e-06,
+      "loss": 0.0445,
+      "step": 2990
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 1.018471360206604,
+      "learning_rate": 4.211590296495957e-06,
+      "loss": 0.0518,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9632,
+      "grad_norm": 0.7452988624572754,
+      "learning_rate": 3.8746630727762805e-06,
+      "loss": 0.0474,
+      "step": 3010
+    },
+    {
+      "epoch": 0.9664,
+      "grad_norm": 0.8698496222496033,
+      "learning_rate": 3.5377358490566038e-06,
+      "loss": 0.0451,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9696,
+      "grad_norm": 2.03416109085083,
+      "learning_rate": 3.200808625336928e-06,
+      "loss": 0.043,
+      "step": 3030
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 0.6438256502151489,
+      "learning_rate": 2.8638814016172507e-06,
+      "loss": 0.0471,
+      "step": 3040
+    },
+    {
+      "epoch": 0.976,
+      "grad_norm": 0.7598551511764526,
+      "learning_rate": 2.5269541778975744e-06,
+      "loss": 0.0459,
+      "step": 3050
+    },
+    {
+      "epoch": 0.9792,
+      "grad_norm": 1.1387499570846558,
+      "learning_rate": 2.1900269541778976e-06,
+      "loss": 0.0527,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9824,
+      "grad_norm": 0.6096578240394592,
+      "learning_rate": 1.853099730458221e-06,
+      "loss": 0.0394,
+      "step": 3070
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 0.8008269667625427,
+      "learning_rate": 1.5161725067385445e-06,
+      "loss": 0.0382,
+      "step": 3080
+    },
+    {
+      "epoch": 0.9888,
+      "grad_norm": 0.7384421229362488,
+      "learning_rate": 1.179245283018868e-06,
+      "loss": 0.047,
+      "step": 3090
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 0.5688408613204956,
+      "learning_rate": 8.423180592991913e-07,
+      "loss": 0.0453,
+      "step": 3100
+    },
+    {
+      "epoch": 0.9952,
+      "grad_norm": 0.7230699062347412,
+      "learning_rate": 5.053908355795148e-07,
+      "loss": 0.0463,
+      "step": 3110
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 1.020484447479248,
+      "learning_rate": 1.684636118598383e-07,
+      "loss": 0.0425,
+      "step": 3120
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 3125,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-3125/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53ff736f14d5430a9239450432341446201a9d3868cd4cd297d3d117d05db1da
+size 5304

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "gpt2-medium",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_inner": null,
+  "n_layer": 24,
+  "n_positions": 1024,
+  "n_special": 0,
+  "predict_special_tokens": true,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "transformers_version": "4.45.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_logs.csv

@@ -0,0 +1,314 @@
+step,training_loss,grad_norm
+10,22.4391,5.0
+20,18.5398,5.0
+30,18.8691,5.000000476837158
+40,17.9809,5.0
+50,15.009,5.0
+60,14.248,4.999999523162842
+70,11.8696,5.0
+80,9.5315,4.999999046325684
+90,7.9867,4.999999046325684
+100,6.6488,5.0
+110,4.6404,4.999999523162842
+120,4.1932,4.999999523162842
+130,3.4752,4.999999046325684
+140,2.8687,4.999999523162842
+150,2.5653,4.999999523162842
+160,2.2596,4.999999046325684
+170,1.9221,4.999999046325684
+180,1.7448,4.999999523162842
+190,1.6752,4.999999046325684
+200,1.5561,4.960172653198242
+210,1.4908,4.512001991271973
+220,1.2773,4.827399253845215
+230,1.2566,3.5412518978118896
+240,1.3037,4.866581439971924
+250,1.1427,3.3435661792755127
+260,1.19,4.5573577880859375
+270,1.2111,4.2881903648376465
+280,1.1723,3.5418100357055664
+290,1.0575,3.8908731937408447
+300,1.0798,3.236272096633911
+310,1.0962,4.228330135345459
+320,1.0134,4.129443645477295
+330,0.9604,4.085168361663818
+340,1.0164,3.8179402351379395
+350,0.9685,3.624455451965332
+360,0.8864,3.5315566062927246
+370,0.9273,3.7452166080474854
+380,0.8983,4.054487228393555
+390,0.8788,4.036263942718506
+400,0.7646,3.101487874984741
+410,0.8178,3.928171157836914
+420,0.7828,4.1417059898376465
+430,0.8381,4.999999046325684
+440,0.8254,4.596191883087158
+450,0.772,3.0273566246032715
+460,0.7263,2.827469825744629
+470,0.7221,3.2652060985565186
+480,0.702,3.0766632556915283
+490,0.6418,2.8645482063293457
+500,0.7149,3.1364777088165283
+510,0.6353,4.046690940856934
+520,0.6268,3.265176296234131
+530,0.5919,2.857509136199951
+540,0.6128,3.4214327335357666
+550,0.6164,4.073673725128174
+560,0.6125,3.854487895965576
+570,0.5139,3.6991710662841797
+580,0.5112,3.3676042556762695
+590,0.5244,2.7846508026123047
+600,0.4886,3.6226212978363037
+610,0.5436,4.951099395751953
+620,0.4834,3.92474102973938
+630,0.4462,3.5895919799804688
+640,0.4694,3.756981372833252
+650,0.4597,3.021611213684082
+660,0.3989,4.999999046325684
+670,0.4079,4.999999046325684
+680,0.3751,3.6756749153137207
+690,0.3304,2.9354946613311768
+700,0.3414,4.999999046325684
+710,0.3368,4.999999046325684
+720,0.2955,2.68564772605896
+730,0.2871,3.137021780014038
+740,0.305,2.4497194290161133
+750,0.2434,3.228701114654541
+760,0.2763,2.0787534713745117
+770,0.2501,3.1535820960998535
+780,0.2706,1.965580701828003
+790,0.2359,2.9210808277130127
+800,0.2389,3.354069232940674
+810,0.1947,2.353992223739624
+820,0.196,2.1019110679626465
+830,0.2167,2.902089834213257
+840,0.1899,2.7410049438476562
+850,0.1795,2.143695116043091
+860,0.2081,1.853039264678955
+870,0.2033,2.7920002937316895
+880,0.1762,1.7611720561981201
+890,0.1721,2.224700450897217
+900,0.166,1.5112501382827759
+910,0.1605,1.5669995546340942
+920,0.1565,1.366951584815979
+930,0.1562,3.0640170574188232
+940,0.1568,1.766140103340149
+950,0.1554,1.9538131952285767
+960,0.1316,1.3904852867126465
+970,0.1653,1.8764123916625977
+980,0.1299,1.753258228302002
+990,0.1332,1.5121208429336548
+1000,0.1375,1.6741907596588135
+1010,0.1345,1.113685131072998
+1020,0.1346,2.015662431716919
+1030,0.1297,1.538110375404358
+1040,0.1331,1.3940174579620361
+1050,0.1328,2.0724356174468994
+1060,0.1322,1.7255312204360962
+1070,0.1415,2.1497256755828857
+1080,0.1227,1.7581284046173096
+1090,0.1285,1.199498176574707
+1100,0.1204,1.792433500289917
+1110,0.1126,1.1998846530914307
+1120,0.1137,1.4056320190429688
+1130,0.1156,2.3784384727478027
+1140,0.1179,1.422841191291809
+1150,0.1135,1.3304040431976318
+1160,0.1146,1.1918659210205078
+1170,0.1124,1.455103874206543
+1180,0.1016,1.1681839227676392
+1190,0.1067,2.063532829284668
+1200,0.103,2.1005892753601074
+1210,0.1016,1.8993762731552124
+1220,0.1139,1.246466875076294
+1230,0.0944,1.0742267370224
+1240,0.1047,1.6990922689437866
+1250,0.0984,0.9877095818519592
+1260,0.1018,2.03071665763855
+1270,0.099,1.1828073263168335
+1280,0.1119,2.286886692047119
+1290,0.1062,1.35301673412323
+1300,0.0919,1.1716175079345703
+1310,0.0941,1.2347395420074463
+1320,0.0917,1.1255364418029785
+1330,0.0945,1.3256624937057495
+1340,0.104,1.4447101354599
+1350,0.0856,1.0093165636062622
+1360,0.0957,2.2328953742980957
+1370,0.0933,1.4997823238372803
+1380,0.0961,0.8270460367202759
+1390,0.085,1.8531345129013062
+1400,0.0966,1.0742558240890503
+1410,0.0848,1.1224265098571777
+1420,0.0894,1.5263229608535767
+1430,0.0888,1.7091747522354126
+1440,0.0843,1.8716164827346802
+1450,0.0869,0.9060105681419373
+1460,0.079,1.1350743770599365
+1470,0.0835,1.1754982471466064
+1480,0.0884,1.6735552549362183
+1490,0.0756,1.191200613975525
+1500,0.0685,1.3234206438064575
+1510,0.0798,1.2499926090240479
+1520,0.0825,1.8494608402252197
+1530,0.0797,1.1932600736618042
+1540,0.074,0.9455847144126892
+1550,0.077,0.9006221294403076
+1560,0.0802,1.0043009519577026
+1570,0.0793,1.027671456336975
+1580,0.0793,1.51658296585083
+1590,0.0784,1.336788535118103
+1600,0.0706,1.3772467374801636
+1610,0.0781,0.922598123550415
+1620,0.0825,1.441529631614685
+1630,0.0767,0.9344118237495422
+1640,0.0715,2.2413012981414795
+1650,0.0814,1.0326889753341675
+1660,0.0707,1.0714298486709595
+1670,0.0802,1.345158338546753
+1680,0.0734,0.9195489883422852
+1690,0.0744,0.7901739478111267
+1700,0.068,1.0859198570251465
+1710,0.068,1.1353799104690552
+1720,0.0575,0.8822022080421448
+1730,0.0702,0.9976137280464172
+1740,0.0715,2.010540008544922
+1750,0.0714,2.2936131954193115
+1760,0.0718,0.8874325752258301
+1770,0.068,0.8641786575317383
+1780,0.0772,1.0465822219848633
+1790,0.0703,1.080430269241333
+1800,0.0801,1.214148759841919
+1810,0.0634,1.1502673625946045
+1820,0.0592,0.7351061105728149
+1830,0.0654,0.5843839049339294
+1840,0.0622,0.8455338478088379
+1850,0.0665,1.4609434604644775
+1860,0.0614,0.9410058259963989
+1870,0.0622,0.9071024060249329
+1880,0.0645,1.175545334815979
+1890,0.0618,1.1436173915863037
+1900,0.0625,1.0221768617630005
+1910,0.0649,1.6312997341156006
+1920,0.0689,0.963321328163147
+1930,0.0593,1.2729462385177612
+1940,0.0542,1.2127982378005981
+1950,0.0626,0.7582926750183105
+1960,0.0629,1.59578275680542
+1970,0.0595,1.157958745956421
+1980,0.0601,2.2247676849365234
+1990,0.0552,0.6778025031089783
+2000,0.0574,1.1828677654266357
+2010,0.0535,0.7850990295410156
+2020,0.0572,1.0724990367889404
+2030,0.0571,1.0207794904708862
+2040,0.0626,0.7693538665771484
+2050,0.0566,0.7968879342079163
+2060,0.0594,1.2296173572540283
+2070,0.0513,0.8568419218063354
+2080,0.0547,0.9537982940673828
+2090,0.055,0.8022981882095337
+2100,0.0575,1.1287271976470947
+2110,0.0687,1.0856471061706543
+2120,0.0523,0.9026399850845337
+2130,0.0545,0.8722916841506958
+2140,0.054,1.5549736022949219
+2150,0.0536,0.989742636680603
+2160,0.0492,1.279964566230774
+2170,0.0598,0.8057636022567749
+2180,0.0626,1.6911473274230957
+2190,0.0568,0.7519389986991882
+2200,0.0584,1.1423537731170654
+2210,0.0639,1.7910997867584229
+2220,0.0577,0.8631108403205872
+2230,0.0526,0.8344480395317078
+2240,0.0564,1.616515874862671
+2250,0.0551,1.0607733726501465
+2260,0.0473,0.6787971258163452
+2270,0.0569,1.0428708791732788
+2280,0.0533,0.9818703532218933
+2290,0.0553,1.5224767923355103
+2300,0.0532,1.2571505308151245
+2310,0.0487,0.7414546608924866
+2320,0.0557,1.0934925079345703
+2330,0.0494,1.0424625873565674
+2340,0.0577,1.1907474994659424
+2350,0.0538,1.0648528337478638
+2360,0.0506,0.5466381907463074
+2370,0.0499,1.2638976573944092
+2380,0.0518,1.2526659965515137
+2390,0.049,0.7721813917160034
+2400,0.0526,0.6863059997558594
+2410,0.0499,0.7994042634963989
+2420,0.0492,1.0728378295898438
+2430,0.0461,1.2912821769714355
+2440,0.0453,0.9025666117668152
+2450,0.0471,0.6996231079101562
+2460,0.0538,0.7211071252822876
+2470,0.0484,0.6897232532501221
+2480,0.0446,1.3392776250839233
+2490,0.0467,0.5439741015434265
+2500,0.0469,0.6194649934768677
+2510,0.0459,0.7839043736457825
+2520,0.0483,0.9303894639015198
+2530,0.0464,1.0741106271743774
+2540,0.0477,0.9432321190834045
+2550,0.0504,0.8696873188018799
+2560,0.0478,1.1203161478042603
+2570,0.0518,1.0000349283218384
+2580,0.0487,0.7597396373748779
+2590,0.0458,0.6735145449638367
+2600,0.0408,0.7138289213180542
+2610,0.0448,0.944825291633606
+2620,0.0465,0.8002263307571411
+2630,0.0486,1.3529142141342163
+2640,0.05,1.245259404182434
+2650,0.0493,0.767680287361145
+2660,0.051,1.5667990446090698
+2670,0.0503,1.1249333620071411
+2680,0.0404,0.6794225573539734
+2690,0.0466,0.94537752866745
+2700,0.0457,0.9569212794303894
+2710,0.0537,0.7343612313270569
+2720,0.0447,1.2045423984527588
+2730,0.0477,0.7130928635597229
+2740,0.048,1.2449103593826294
+2750,0.044,1.1206499338150024
+2760,0.0611,0.7373448610305786
+2770,0.0478,1.2338485717773438
+2780,0.0415,0.931049108505249
+2790,0.0441,0.818911612033844
+2800,0.0445,0.5560767650604248
+2810,0.0406,0.6225351095199585
+2820,0.0464,1.02946138381958
+2830,0.0508,0.7534985542297363
+2840,0.0479,0.7269020080566406
+2850,0.0487,0.6104754209518433
+2860,0.0434,0.7579702734947205
+2870,0.0447,0.7916242480278015
+2880,0.0468,0.9057757258415222
+2890,0.0488,0.6375102400779724
+2900,0.0554,1.7158457040786743
+2910,0.0453,0.8057986497879028
+2920,0.0416,0.8072325587272644
+2930,0.046,0.6157782673835754
+2940,0.0491,0.5398648977279663
+2950,0.0469,1.032221794128418
+2960,0.0441,0.8197404742240906
+2970,0.044,0.7197135090827942
+2980,0.046,0.8980445861816406
+2990,0.0445,0.7673968076705933
+3000,0.0518,1.018471360206604
+3010,0.0474,0.7452988624572754
+3020,0.0451,0.8698496222496033
+3030,0.043,2.03416109085083
+3040,0.0471,0.6438256502151489
+3050,0.0459,0.7598551511764526
+3060,0.0527,1.1387499570846558
+3070,0.0394,0.6096578240394592
+3080,0.0382,0.8008269667625427
+3090,0.047,0.7384421229362488
+3100,0.0453,0.5688408613204956
+3110,0.0463,0.7230699062347412
+3120,0.0425,1.020484447479248
+3125,nan,nan