multi-nl2-nn2.log

USING CUDA
USING CUDA
Manual seed for pytorch: 1427435741
Hyperparameters:
attention_dropout 0.2
batch_cky False
bert_do_lower_case True
bert_model 'bert-multilingual-cased'
bert_transliterate ''
char_lstm_input_dropout 0.2
clip_grad_norm 0.0
d_char_emb 32
d_ff 2048
d_kv 64
d_label_hidden 256
d_model 1024
d_tag_hidden 256
elmo_dropout 0.5
embedding_dropout 0.0
integration 'hard'
label_weights False
learning_rate 5e-05
learning_rate_warmup_steps 160
max_consecutive_decays 3
max_len_dev 0
max_len_train 0
metric 'l2'
morpho_emb_dropout 0.2
no_mlp False
num_heads 8
num_layers 2
num_layers_position_only 0
partitioned True
predict_tags False
relu_dropout 0.1
residual_dropout 0.2
sentence_max_len 300
step_decay True
step_decay_factor 0.5
step_decay_patience 5
tag_emb_dropout 0.2
tag_loss_scale 5.0
timing_dropout 0.0
use_bert True
use_bert_only False
use_chars_lstm False
use_elmo False
use_label_weights False
use_tags False
use_words False
word_emb_dropout 0.4
zero_empty False
Loading training trees from data/02-21.10way.clean...
Loaded 39,832 training examples.
Loading development trees from data/22.auto.clean...
Loaded 1,700 development examples.
Processing trees for training...
Constructing vocabularies...
Initializing model...
Loading parameters from models/en_multibert_empty_l2_dev=94.94.pt
Using gpu 0 for index
Initializing optimizer...
Training...
epoch 1 batch 1/1,245 processed 32 batch-loss 63.5120 grad-norm 552.8393 epoch-elapsed 0h00m02s total-elapsed 0h00m02s
epoch 1 batch 2/1,245 processed 64 batch-loss 89.9655 grad-norm 651.3477 epoch-elapsed 0h00m06s total-elapsed 0h00m06s
epoch 1 batch 3/1,245 processed 96 batch-loss 79.7717 grad-norm 513.5317 epoch-elapsed 0h00m09s total-elapsed 0h00m09s
epoch 1 batch 4/1,245 processed 128 batch-loss 92.0810 grad-norm 734.2422 epoch-elapsed 0h00m12s total-elapsed 0h00m12s
epoch 1 batch 5/1,245 processed 160 batch-loss 65.9747 grad-norm 469.0768 epoch-elapsed 0h00m15s total-elapsed 0h00m15s
epoch 1 batch 6/1,245 processed 192 batch-loss 69.6999 grad-norm 636.3662 epoch-elapsed 0h00m18s total-elapsed 0h00m18s
epoch 1 batch 7/1,245 processed 224 batch-loss 77.6937 grad-norm 476.5899 epoch-elapsed 0h00m20s total-elapsed 0h00m20s
epoch 1 batch 8/1,245 processed 256 batch-loss 52.6241 grad-norm 496.3018 epoch-elapsed 0h00m22s total-elapsed 0h00m22s
epoch 1 batch 9/1,245 processed 288 batch-loss 58.3291 grad-norm 565.6784 epoch-elapsed 0h00m25s total-elapsed 0h00m25s
epoch 1 batch 10/1,245 processed 320 batch-loss 57.2188 grad-norm 343.2288 epoch-elapsed 0h00m27s total-elapsed 0h00m27s
epoch 1 batch 11/1,245 processed 352 batch-loss 60.2387 grad-norm 385.6425 epoch-elapsed 0h00m30s total-elapsed 0h00m30s
epoch 1 batch 12/1,245 processed 384 batch-loss 74.5435 grad-norm 503.3886 epoch-elapsed 0h00m33s total-elapsed 0h00m33s
epoch 1 batch 13/1,245 processed 416 batch-loss 66.2096 grad-norm 534.5345 epoch-elapsed 0h00m35s total-elapsed 0h00m35s
epoch 1 batch 14/1,245 processed 448 batch-loss 47.6130 grad-norm 415.1659 epoch-elapsed 0h00m37s total-elapsed 0h00m37s
epoch 1 batch 15/1,245 processed 480 batch-loss 63.0688 grad-norm 404.9516 epoch-elapsed 0h00m40s total-elapsed 0h00m40s
epoch 1 batch 16/1,245 processed 512 batch-loss 54.2146 grad-norm 429.3789 epoch-elapsed 0h00m43s total-elapsed 0h00m43s
epoch 1 batch 17/1,245 processed 544 batch-loss 44.0166 grad-norm 390.0671 epoch-elapsed 0h00m45s total-elapsed 0h00m45s
epoch 1 batch 18/1,245 processed 576 batch-loss 66.8780 grad-norm 515.7155 epoch-elapsed 0h00m48s total-elapsed 0h00m48s
epoch 1 batch 19/1,245 processed 608 batch-loss 59.4198 grad-norm 409.2954 epoch-elapsed 0h00m51s total-elapsed 0h00m51s
epoch 1 batch 20/1,245 processed 640 batch-loss 63.3942 grad-norm 500.9272 epoch-elapsed 0h00m54s total-elapsed 0h00m54s
epoch 1 batch 21/1,245 processed 672 batch-loss 61.1461 grad-norm 440.6756 epoch-elapsed 0h00m57s total-elapsed 0h00m57s
epoch 1 batch 22/1,245 processed 704 batch-loss 57.6342 grad-norm 472.9212 epoch-elapsed 0h00m59s total-elapsed 0h00m59s
epoch 1 batch 23/1,245 processed 736 batch-loss 47.6893 grad-norm 356.4618 epoch-elapsed 0h01m02s total-elapsed 0h01m02s
epoch 1 batch 24/1,245 processed 768 batch-loss 69.6937 grad-norm 448.2104 epoch-elapsed 0h01m05s total-elapsed 0h01m05s
epoch 1 batch 25/1,245 processed 800 batch-loss 67.7786 grad-norm 480.1780 epoch-elapsed 0h01m09s total-elapsed 0h01m09s
epoch 1 batch 26/1,245 processed 832 batch-loss 52.6160 grad-norm 321.6268 epoch-elapsed 0h01m11s total-elapsed 0h01m11s
epoch 1 batch 27/1,245 processed 864 batch-loss 77.9268 grad-norm 511.1043 epoch-elapsed 0h01m15s total-elapsed 0h01m15s
epoch 1 batch 28/1,245 processed 896 batch-loss 50.6112 grad-norm 415.2423 epoch-elapsed 0h01m17s total-elapsed 0h01m17s
epoch 1 batch 29/1,245 processed 928 batch-loss 51.8887 grad-norm 377.4879 epoch-elapsed 0h01m20s total-elapsed 0h01m20s
epoch 1 batch 30/1,245 processed 960 batch-loss 56.9099 grad-norm 415.5174 epoch-elapsed 0h01m23s total-elapsed 0h01m23s
epoch 1 batch 31/1,245 processed 992 batch-loss 38.9914 grad-norm 376.9804 epoch-elapsed 0h01m25s total-elapsed 0h01m25s
epoch 1 batch 32/1,245 processed 1,024 batch-loss 46.4296 grad-norm 263.3615 epoch-elapsed 0h01m27s total-elapsed 0h01m27s
epoch 1 batch 33/1,245 processed 1,056 batch-loss 52.0367 grad-norm 275.7291 epoch-elapsed 0h01m29s total-elapsed 0h01m29s
epoch 1 batch 34/1,245 processed 1,088 batch-loss 59.6011 grad-norm 423.7762 epoch-elapsed 0h01m33s total-elapsed 0h01m33s
epoch 1 batch 35/1,245 processed 1,120 batch-loss 54.7555 grad-norm 414.2780 epoch-elapsed 0h01m35s total-elapsed 0h01m35s
epoch 1 batch 36/1,245 processed 1,152 batch-loss 46.7118 grad-norm 344.4381 epoch-elapsed 0h01m38s total-elapsed 0h01m38s
epoch 1 batch 37/1,245 processed 1,184 batch-loss 54.4260 grad-norm 272.5578 epoch-elapsed 0h01m40s total-elapsed 0h01m40s
epoch 1 batch 38/1,245 processed 1,216 batch-loss 52.6105 grad-norm 383.4745 epoch-elapsed 0h01m44s total-elapsed 0h01m44s
epoch 1 batch 39/1,245 processed 1,248 batch-loss 50.8064 grad-norm 279.4853 epoch-elapsed 0h01m46s total-elapsed 0h01m46s
epoch 1 batch 40/1,245 processed 1,280 batch-loss 59.0239 grad-norm 312.7569 epoch-elapsed 0h01m48s total-elapsed 0h01m48s
epoch 1 batch 41/1,245 processed 1,312 batch-loss 53.1670 grad-norm 318.1699 epoch-elapsed 0h01m51s total-elapsed 0h01m51s
epoch 1 batch 42/1,245 processed 1,344 batch-loss 53.1772 grad-norm 275.7435 epoch-elapsed 0h01m54s total-elapsed 0h01m54s
epoch 1 batch 43/1,245 processed 1,376 batch-loss 73.3996 grad-norm 303.8220 epoch-elapsed 0h02m00s total-elapsed 0h02m00s
epoch 1 batch 44/1,245 processed 1,408 batch-loss 60.3037 grad-norm 274.9060 epoch-elapsed 0h02m02s total-elapsed 0h02m02s
epoch 1 batch 45/1,245 processed 1,440 batch-loss 44.9248 grad-norm 711.2790 epoch-elapsed 0h02m05s total-elapsed 0h02m05s
epoch 1 batch 46/1,245 processed 1,472 batch-loss 59.5276 grad-norm 348.4368 epoch-elapsed 0h02m08s total-elapsed 0h02m08s
epoch 1 batch 47/1,245 processed 1,504 batch-loss 58.4336 grad-norm 402.6082 epoch-elapsed 0h02m10s total-elapsed 0h02m10s
epoch 1 batch 48/1,245 processed 1,536 batch-loss 54.8541 grad-norm 345.7608 epoch-elapsed 0h02m14s total-elapsed 0h02m14s
epoch 1 batch 49/1,245 processed 1,568 batch-loss 67.6469 grad-norm 349.6040 epoch-elapsed 0h02m19s total-elapsed 0h02m19s
epoch 1 batch 50/1,245 processed 1,600 batch-loss 51.7965 grad-norm 272.3436 epoch-elapsed 0h02m22s total-elapsed 0h02m22s
epoch 1 batch 51/1,245 processed 1,632 batch-loss 42.6608 grad-norm 338.5732 epoch-elapsed 0h02m24s total-elapsed 0h02m24s
epoch 1 batch 52/1,245 processed 1,664 batch-loss 58.2224 grad-norm 263.2672 epoch-elapsed 0h02m28s total-elapsed 0h02m28s
epoch 1 batch 53/1,245 processed 1,696 batch-loss 59.8471 grad-norm 305.5941 epoch-elapsed 0h02m33s total-elapsed 0h02m33s
epoch 1 batch 54/1,245 processed 1,728 batch-loss 54.1996 grad-norm 255.5483 epoch-elapsed 0h02m36s total-elapsed 0h02m36s
epoch 1 batch 55/1,245 processed 1,760 batch-loss 44.7625 grad-norm 266.3612 epoch-elapsed 0h02m39s total-elapsed 0h02m39s
epoch 1 batch 56/1,245 processed 1,792 batch-loss 43.1432 grad-norm 549.7106 epoch-elapsed 0h02m41s total-elapsed 0h02m41s
epoch 1 batch 57/1,245 processed 1,824 batch-loss 40.1677 grad-norm 351.7434 epoch-elapsed 0h02m44s total-elapsed 0h02m44s
epoch 1 batch 58/1,245 processed 1,856 batch-loss 41.0305 grad-norm 346.1233 epoch-elapsed 0h02m46s total-elapsed 0h02m46s
epoch 1 batch 59/1,245 processed 1,888 batch-loss 36.5260 grad-norm 196.0312 epoch-elapsed 0h02m48s total-elapsed 0h02m48s
epoch 1 batch 60/1,245 processed 1,920 batch-loss 50.9581 grad-norm 485.1127 epoch-elapsed 0h02m51s total-elapsed 0h02m51s
epoch 1 batch 61/1,245 processed 1,952 batch-loss 40.7929 grad-norm 185.8428 epoch-elapsed 0h02m53s total-elapsed 0h02m53s
epoch 1 batch 62/1,245 processed 1,984 batch-loss 37.5425 grad-norm 328.1523 epoch-elapsed 0h02m55s total-elapsed 0h02m55s
epoch 1 batch 63/1,245 processed 2,016 batch-loss 44.7660 grad-norm 227.1669 epoch-elapsed 0h02m58s total-elapsed 0h02m58s
epoch 1 batch 64/1,245 processed 2,048 batch-loss 38.3840 grad-norm 197.4038 epoch-elapsed 0h03m00s total-elapsed 0h03m00s
epoch 1 batch 65/1,245 processed 2,080 batch-loss 45.0887 grad-norm 204.3456 epoch-elapsed 0h03m02s total-elapsed 0h03m02s
epoch 1 batch 66/1,245 processed 2,112 batch-loss 46.3293 grad-norm 202.3097 epoch-elapsed 0h03m05s total-elapsed 0h03m05s
epoch 1 batch 67/1,245 processed 2,144 batch-loss 46.3482 grad-norm 251.5533 epoch-elapsed 0h03m09s total-elapsed 0h03m09s
epoch 1 batch 68/1,245 processed 2,176 batch-loss 48.9302 grad-norm 212.2068 epoch-elapsed 0h03m12s total-elapsed 0h03m12s
epoch 1 batch 69/1,245 processed 2,208 batch-loss 44.5943 grad-norm 369.8784 epoch-elapsed 0h03m15s total-elapsed 0h03m15s
epoch 1 batch 70/1,245 processed 2,240 batch-loss 47.9706 grad-norm 208.7859 epoch-elapsed 0h03m17s total-elapsed 0h03m17s
epoch 1 batch 71/1,245 processed 2,272 batch-loss 40.0387 grad-norm 305.7398 epoch-elapsed 0h03m20s total-elapsed 0h03m20s
epoch 1 batch 72/1,245 processed 2,304 batch-loss 38.1265 grad-norm 165.5051 epoch-elapsed 0h03m22s total-elapsed 0h03m22s
epoch 1 batch 73/1,245 processed 2,336 batch-loss 40.9030 grad-norm 238.5774 epoch-elapsed 0h03m25s total-elapsed 0h03m25s
epoch 1 batch 74/1,245 processed 2,368 batch-loss 39.2019 grad-norm 216.3151 epoch-elapsed 0h03m28s total-elapsed 0h03m28s
epoch 1 batch 75/1,245 processed 2,400 batch-loss 46.5692 grad-norm 205.1435 epoch-elapsed 0h03m31s total-elapsed 0h03m31s
epoch 1 batch 76/1,245 processed 2,432 batch-loss 41.2039 grad-norm 367.0629 epoch-elapsed 0h03m35s total-elapsed 0h03m35s
epoch 1 batch 77/1,245 processed 2,464 batch-loss 48.1440 grad-norm 264.3229 epoch-elapsed 0h03m39s total-elapsed 0h03m39s
epoch 1 batch 78/1,245 processed 2,496 batch-loss 35.6898 grad-norm 263.2270 epoch-elapsed 0h03m42s total-elapsed 0h03m42s
epoch 1 batch 79/1,245 processed 2,528 batch-loss 40.5352 grad-norm 307.0469 epoch-elapsed 0h03m44s total-elapsed 0h03m44s
epoch 1 batch 80/1,245 processed 2,560 batch-loss 35.0213 grad-norm 155.6833 epoch-elapsed 0h03m47s total-elapsed 0h03m47s
epoch 1 batch 81/1,245 processed 2,592 batch-loss 36.6507 grad-norm 205.5800 epoch-elapsed 0h03m50s total-elapsed 0h03m50s
epoch 1 batch 82/1,245 processed 2,624 batch-loss 33.7812 grad-norm 270.0532 epoch-elapsed 0h03m52s total-elapsed 0h03m52s
epoch 1 batch 83/1,245 processed 2,656 batch-loss 37.7353 grad-norm 288.6348 epoch-elapsed 0h03m54s total-elapsed 0h03m54s
epoch 1 batch 84/1,245 processed 2,688 batch-loss 34.4940 grad-norm 198.7269 epoch-elapsed 0h03m57s total-elapsed 0h03m57s
epoch 1 batch 85/1,245 processed 2,720 batch-loss 33.2568 grad-norm 195.5450 epoch-elapsed 0h03m59s total-elapsed 0h03m59s
epoch 1 batch 86/1,245 processed 2,752 batch-loss 33.7383 grad-norm 172.7423 epoch-elapsed 0h04m03s total-elapsed 0h04m03s
epoch 1 batch 87/1,245 processed 2,784 batch-loss 38.8420 grad-norm 206.5452 epoch-elapsed 0h04m06s total-elapsed 0h04m06s
epoch 1 batch 88/1,245 processed 2,816 batch-loss 33.6041 grad-norm 253.7655 epoch-elapsed 0h04m09s total-elapsed 0h04m09s
epoch 1 batch 89/1,245 processed 2,848 batch-loss 34.8962 grad-norm 319.7504 epoch-elapsed 0h04m12s total-elapsed 0h04m12s
epoch 1 batch 90/1,245 processed 2,880 batch-loss 38.0831 grad-norm 205.1577 epoch-elapsed 0h04m15s total-elapsed 0h04m15s
epoch 1 batch 91/1,245 processed 2,912 batch-loss 38.1175 grad-norm 197.0517 epoch-elapsed 0h04m18s total-elapsed 0h04m18s
epoch 1 batch 92/1,245 processed 2,944 batch-loss 31.1754 grad-norm 207.4238 epoch-elapsed 0h04m21s total-elapsed 0h04m21s
epoch 1 batch 93/1,245 processed 2,976 batch-loss 27.0633 grad-norm 234.2413 epoch-elapsed 0h04m24s total-elapsed 0h04m24s
epoch 1 batch 94/1,245 processed 3,008 batch-loss 32.3039 grad-norm 211.7322 epoch-elapsed 0h04m26s total-elapsed 0h04m26s
epoch 1 batch 95/1,245 processed 3,040 batch-loss 29.8056 grad-norm 227.9838 epoch-elapsed 0h04m29s total-elapsed 0h04m29s
epoch 1 batch 96/1,245 processed 3,072 batch-loss 33.8411 grad-norm 259.0917 epoch-elapsed 0h04m32s total-elapsed 0h04m32s
epoch 1 batch 97/1,245 processed 3,104 batch-loss 28.8645 grad-norm 484.2102 epoch-elapsed 0h04m34s total-elapsed 0h04m34s
epoch 1 batch 98/1,245 processed 3,136 batch-loss 34.4320 grad-norm 309.1790 epoch-elapsed 0h04m37s total-elapsed 0h04m37s
epoch 1 batch 99/1,245 processed 3,168 batch-loss 33.4138 grad-norm 256.4947 epoch-elapsed 0h04m40s total-elapsed 0h04m40s
epoch 1 batch 100/1,245 processed 3,200 batch-loss 33.0292 grad-norm 163.2962 epoch-elapsed 0h04m43s total-elapsed 0h04m43s
epoch 1 batch 101/1,245 processed 3,232 batch-loss 36.9587 grad-norm 226.3127 epoch-elapsed 0h04m47s total-elapsed 0h04m47s
epoch 1 batch 102/1,245 processed 3,264 batch-loss 33.2831 grad-norm 233.5955 epoch-elapsed 0h04m50s total-elapsed 0h04m50s
epoch 1 batch 103/1,245 processed 3,296 batch-loss 35.6863 grad-norm 209.0562 epoch-elapsed 0h04m54s total-elapsed 0h04m54s
epoch 1 batch 104/1,245 processed 3,328 batch-loss 28.1802 grad-norm 110.6291 epoch-elapsed 0h04m56s total-elapsed 0h04m56s
epoch 1 batch 105/1,245 processed 3,360 batch-loss 34.1203 grad-norm 211.3528 epoch-elapsed 0h04m59s total-elapsed 0h04m59s
epoch 1 batch 106/1,245 processed 3,392 batch-loss 24.4220 grad-norm 129.2499 epoch-elapsed 0h05m01s total-elapsed 0h05m01s
epoch 1 batch 107/1,245 processed 3,424 batch-loss 28.6813 grad-norm 241.9950 epoch-elapsed 0h05m04s total-elapsed 0h05m04s
epoch 1 batch 108/1,245 processed 3,456 batch-loss 31.4063 grad-norm 228.1232 epoch-elapsed 0h05m08s total-elapsed 0h05m08s
epoch 1 batch 109/1,245 processed 3,488 batch-loss 28.0648 grad-norm 151.7690 epoch-elapsed 0h05m11s total-elapsed 0h05m11s
epoch 1 batch 110/1,245 processed 3,520 batch-loss 29.3143 grad-norm 197.4859 epoch-elapsed 0h05m15s total-elapsed 0h05m15s
epoch 1 batch 111/1,245 processed 3,552 batch-loss 24.8180 grad-norm 110.8323 epoch-elapsed 0h05m17s total-elapsed 0h05m17s
epoch 1 batch 112/1,245 processed 3,584 batch-loss 24.0707 grad-norm 231.9842 epoch-elapsed 0h05m20s total-elapsed 0h05m20s
epoch 1 batch 113/1,245 processed 3,616 batch-loss 27.9297 grad-norm 404.1207 epoch-elapsed 0h05m23s total-elapsed 0h05m23s
epoch 1 batch 114/1,245 processed 3,648 batch-loss 21.8561 grad-norm 166.2111 epoch-elapsed 0h05m25s total-elapsed 0h05m25s
epoch 1 batch 115/1,245 processed 3,680 batch-loss 17.6555 grad-norm 145.7969 epoch-elapsed 0h05m28s total-elapsed 0h05m28s
epoch 1 batch 116/1,245 processed 3,712 batch-loss 23.2752 grad-norm 182.3490 epoch-elapsed 0h05m31s total-elapsed 0h05m31s
epoch 1 batch 117/1,245 processed 3,744 batch-loss 25.6976 grad-norm 129.2352 epoch-elapsed 0h05m35s total-elapsed 0h05m35s
epoch 1 batch 118/1,245 processed 3,776 batch-loss 23.8284 grad-norm 298.2215 epoch-elapsed 0h05m38s total-elapsed 0h05m38s
epoch 1 batch 119/1,245 processed 3,808 batch-loss 25.2065 grad-norm 189.4226 epoch-elapsed 0h05m41s total-elapsed 0h05m41s
epoch 1 batch 120/1,245 processed 3,840 batch-loss 25.9821 grad-norm 134.0704 epoch-elapsed 0h05m45s total-elapsed 0h05m45s
epoch 1 batch 121/1,245 processed 3,872 batch-loss 24.2337 grad-norm 248.2298 epoch-elapsed 0h05m49s total-elapsed 0h05m49s
epoch 1 batch 122/1,245 processed 3,904 batch-loss 18.5515 grad-norm 210.1929 epoch-elapsed 0h05m51s total-elapsed 0h05m51s
epoch 1 batch 123/1,245 processed 3,936 batch-loss 23.8590 grad-norm 119.7483 epoch-elapsed 0h05m55s total-elapsed 0h05m55s
epoch 1 batch 124/1,245 processed 3,968 batch-loss 25.9553 grad-norm 159.8803 epoch-elapsed 0h05m59s total-elapsed 0h05m59s
epoch 1 batch 125/1,245 processed 4,000 batch-loss 21.0693 grad-norm 150.4415 epoch-elapsed 0h06m02s total-elapsed 0h06m02s
epoch 1 batch 126/1,245 processed 4,032 batch-loss 25.5406 grad-norm 152.2240 epoch-elapsed 0h06m07s total-elapsed 0h06m07s
epoch 1 batch 127/1,245 processed 4,064 batch-loss 19.3758 grad-norm 170.4233 epoch-elapsed 0h06m10s total-elapsed 0h06m10s
epoch 1 batch 128/1,245 processed 4,096 batch-loss 19.9198 grad-norm 152.3505 epoch-elapsed 0h06m14s total-elapsed 0h06m14s
epoch 1 batch 129/1,245 processed 4,128 batch-loss 13.4468 grad-norm 94.2665 epoch-elapsed 0h06m17s total-elapsed 0h06m17s
epoch 1 batch 130/1,245 processed 4,160 batch-loss 23.5399 grad-norm 151.4809 epoch-elapsed 0h06m21s total-elapsed 0h06m21s
epoch 1 batch 131/1,245 processed 4,192 batch-loss 16.7629 grad-norm 151.0880 epoch-elapsed 0h06m24s total-elapsed 0h06m24s
epoch 1 batch 132/1,245 processed 4,224 batch-loss 22.1688 grad-norm 196.1520 epoch-elapsed 0h06m27s total-elapsed 0h06m27s
epoch 1 batch 133/1,245 processed 4,256 batch-loss 18.6556 grad-norm 125.5431 epoch-elapsed 0h06m31s total-elapsed 0h06m31s
epoch 1 batch 134/1,245 processed 4,288 batch-loss 25.5897 grad-norm 244.7853 epoch-elapsed 0h06m36s total-elapsed 0h06m36s
epoch 1 batch 135/1,245 processed 4,320 batch-loss 19.6252 grad-norm 155.5932 epoch-elapsed 0h06m41s total-elapsed 0h06m41s
epoch 1 batch 136/1,245 processed 4,352 batch-loss 18.6204 grad-norm 201.0644 epoch-elapsed 0h06m46s total-elapsed 0h06m46s
epoch 1 batch 137/1,245 processed 4,384 batch-loss 21.5170 grad-norm 140.5555 epoch-elapsed 0h06m50s total-elapsed 0h06m50s
epoch 1 batch 138/1,245 processed 4,416 batch-loss 19.0773 grad-norm 166.8278 epoch-elapsed 0h06m54s total-elapsed 0h06m54s
epoch 1 batch 139/1,245 processed 4,448 batch-loss 22.3640 grad-norm 144.9510 epoch-elapsed 0h06m58s total-elapsed 0h06m58s
epoch 1 batch 140/1,245 processed 4,480 batch-loss 14.6264 grad-norm 95.1458 epoch-elapsed 0h07m02s total-elapsed 0h07m02s
epoch 1 batch 141/1,245 processed 4,512 batch-loss 24.6130 grad-norm 167.8579 epoch-elapsed 0h07m07s total-elapsed 0h07m07s
epoch 1 batch 142/1,245 processed 4,544 batch-loss 25.7318 grad-norm 325.6138 epoch-elapsed 0h07m14s total-elapsed 0h07m14s
epoch 1 batch 143/1,245 processed 4,576 batch-loss 21.1931 grad-norm 210.1026 epoch-elapsed 0h07m19s total-elapsed 0h07m19s
epoch 1 batch 144/1,245 processed 4,608 batch-loss 18.9771 grad-norm 145.7504 epoch-elapsed 0h07m23s total-elapsed 0h07m23s
epoch 1 batch 145/1,245 processed 4,640 batch-loss 14.6223 grad-norm 103.4032 epoch-elapsed 0h07m28s total-elapsed 0h07m28s
epoch 1 batch 146/1,245 processed 4,672 batch-loss 21.6930 grad-norm 275.6725 epoch-elapsed 0h07m31s total-elapsed 0h07m31s
epoch 1 batch 147/1,245 processed 4,704 batch-loss 20.6461 grad-norm 157.9639 epoch-elapsed 0h07m35s total-elapsed 0h07m35s
epoch 1 batch 148/1,245 processed 4,736 batch-loss 22.9895 grad-norm 182.3783 epoch-elapsed 0h07m39s total-elapsed 0h07m39s
epoch 1 batch 149/1,245 processed 4,768 batch-loss 16.3936 grad-norm 147.2051 epoch-elapsed 0h07m43s total-elapsed 0h07m43s
epoch 1 batch 150/1,245 processed 4,800 batch-loss 21.2104 grad-norm 177.4574 epoch-elapsed 0h07m47s total-elapsed 0h07m47s
epoch 1 batch 151/1,245 processed 4,832 batch-loss 11.6629 grad-norm 63.0693 epoch-elapsed 0h07m50s total-elapsed 0h07m50s
epoch 1 batch 152/1,245 processed 4,864 batch-loss 26.1897 grad-norm 357.6483 epoch-elapsed 0h07m55s total-elapsed 0h07m55s
epoch 1 batch 153/1,245 processed 4,896 batch-loss 17.9906 grad-norm 111.7676 epoch-elapsed 0h07m58s total-elapsed 0h07m58s
epoch 1 batch 154/1,245 processed 4,928 batch-loss 17.2070 grad-norm 124.8113 epoch-elapsed 0h08m01s total-elapsed 0h08m01s
epoch 1 batch 155/1,245 processed 4,960 batch-loss 12.3734 grad-norm 291.6047 epoch-elapsed 0h08m04s total-elapsed 0h08m04s
epoch 1 batch 156/1,245 processed 4,992 batch-loss 18.3358 grad-norm 121.8932 epoch-elapsed 0h08m08s total-elapsed 0h08m08s
epoch 1 batch 157/1,245 processed 5,024 batch-loss 12.9581 grad-norm 79.3691 epoch-elapsed 0h08m12s total-elapsed 0h08m12s
epoch 1 batch 158/1,245 processed 5,056 batch-loss 11.9006 grad-norm 102.9527 epoch-elapsed 0h08m15s total-elapsed 0h08m15s
epoch 1 batch 159/1,245 processed 5,088 batch-loss 13.5320 grad-norm 173.7282 epoch-elapsed 0h08m19s total-elapsed 0h08m19s
epoch 1 batch 160/1,245 processed 5,120 batch-loss 13.1755 grad-norm 103.2987 epoch-elapsed 0h08m22s total-elapsed 0h08m22s
epoch 1 batch 161/1,245 processed 5,152 batch-loss 16.8687 grad-norm 138.8387 epoch-elapsed 0h08m27s total-elapsed 0h08m27s
epoch 1 batch 162/1,245 processed 5,184 batch-loss 13.1085 grad-norm 131.0171 epoch-elapsed 0h08m30s total-elapsed 0h08m30s
epoch 1 batch 163/1,245 processed 5,216 batch-loss 14.5898 grad-norm 143.4820 epoch-elapsed 0h08m34s total-elapsed 0h08m34s
epoch 1 batch 164/1,245 processed 5,248 batch-loss 15.1701 grad-norm 110.9446 epoch-elapsed 0h08m37s total-elapsed 0h08m37s
epoch 1 batch 165/1,245 processed 5,280 batch-loss 22.4564 grad-norm 431.0985 epoch-elapsed 0h08m42s total-elapsed 0h08m42s
epoch 1 batch 166/1,245 processed 5,312 batch-loss 14.5435 grad-norm 123.1989 epoch-elapsed 0h08m46s total-elapsed 0h08m46s
epoch 1 batch 167/1,245 processed 5,344 batch-loss 16.5837 grad-norm 101.4268 epoch-elapsed 0h08m50s total-elapsed 0h08m50s
epoch 1 batch 168/1,245 processed 5,376 batch-loss 19.6780 grad-norm 198.9094 epoch-elapsed 0h08m55s total-elapsed 0h08m55s
epoch 1 batch 169/1,245 processed 5,408 batch-loss 16.4869 grad-norm 245.4905 epoch-elapsed 0h08m59s total-elapsed 0h08m59s
epoch 1 batch 170/1,245 processed 5,440 batch-loss 13.7733 grad-norm 224.6185 epoch-elapsed 0h09m03s total-elapsed 0h09m03s
epoch 1 batch 171/1,245 processed 5,472 batch-loss 16.5395 grad-norm 257.7124 epoch-elapsed 0h09m07s total-elapsed 0h09m07s
epoch 1 batch 172/1,245 processed 5,504 batch-loss 11.4310 grad-norm 66.4915 epoch-elapsed 0h09m11s total-elapsed 0h09m11s
epoch 1 batch 173/1,245 processed 5,536 batch-loss 15.1180 grad-norm 176.1107 epoch-elapsed 0h09m14s total-elapsed 0h09m14s
epoch 1 batch 174/1,245 processed 5,568 batch-loss 17.0000 grad-norm 104.4930 epoch-elapsed 0h09m19s total-elapsed 0h09m19s
epoch 1 batch 175/1,245 processed 5,600 batch-loss 13.6503 grad-norm 112.3009 epoch-elapsed 0h09m23s total-elapsed 0h09m23s
epoch 1 batch 176/1,245 processed 5,632 batch-loss 20.2924 grad-norm 290.8586 epoch-elapsed 0h09m28s total-elapsed 0h09m28s
epoch 1 batch 177/1,245 processed 5,664 batch-loss 13.0035 grad-norm 78.2012 epoch-elapsed 0h09m32s total-elapsed 0h09m32s
epoch 1 batch 178/1,245 processed 5,696 batch-loss 18.9282 grad-norm 175.7070 epoch-elapsed 0h09m36s total-elapsed 0h09m36s
epoch 1 batch 179/1,245 processed 5,728 batch-loss 17.7927 grad-norm 194.8097 epoch-elapsed 0h09m40s total-elapsed 0h09m40s
epoch 1 batch 180/1,245 processed 5,760 batch-loss 10.1778 grad-norm 99.3619 epoch-elapsed 0h09m43s total-elapsed 0h09m43s
epoch 1 batch 181/1,245 processed 5,792 batch-loss 12.3820 grad-norm 98.5705 epoch-elapsed 0h09m46s total-elapsed 0h09m46s
epoch 1 batch 182/1,245 processed 5,824 batch-loss 16.0264 grad-norm 155.6436 epoch-elapsed 0h09m50s total-elapsed 0h09m50s
epoch 1 batch 183/1,245 processed 5,856 batch-loss 10.9196 grad-norm 92.2641 epoch-elapsed 0h09m53s total-elapsed 0h09m53s
epoch 1 batch 184/1,245 processed 5,888 batch-loss 12.3567 grad-norm 89.6368 epoch-elapsed 0h09m56s total-elapsed 0h09m56s
epoch 1 batch 185/1,245 processed 5,920 batch-loss 13.8782 grad-norm 184.5461 epoch-elapsed 0h10m00s total-elapsed 0h10m00s
epoch 1 batch 186/1,245 processed 5,952 batch-loss 15.8205 grad-norm 101.1641 epoch-elapsed 0h10m04s total-elapsed 0h10m04s
epoch 1 batch 187/1,245 processed 5,984 batch-loss 12.2147 grad-norm 86.9003 epoch-elapsed 0h10m07s total-elapsed 0h10m07s
epoch 1 batch 188/1,245 processed 6,016 batch-loss 10.6296 grad-norm 77.1110 epoch-elapsed 0h10m10s total-elapsed 0h10m10s
epoch 1 batch 189/1,245 processed 6,048 batch-loss 16.0324 grad-norm 204.7502 epoch-elapsed 0h10m14s total-elapsed 0h10m14s
epoch 1 batch 190/1,245 processed 6,080 batch-loss 17.8701 grad-norm 161.4505 epoch-elapsed 0h10m18s total-elapsed 0h10m18s
epoch 1 batch 191/1,245 processed 6,112 batch-loss 18.1480 grad-norm 179.1355 epoch-elapsed 0h10m22s total-elapsed 0h10m22s
epoch 1 batch 192/1,245 processed 6,144 batch-loss 12.0259 grad-norm 167.2492 epoch-elapsed 0h10m25s total-elapsed 0h10m25s
epoch 1 batch 193/1,245 processed 6,176 batch-loss 10.5317 grad-norm 175.3362 epoch-elapsed 0h10m28s total-elapsed 0h10m28s
epoch 1 batch 194/1,245 processed 6,208 batch-loss 15.9194 grad-norm 95.9158 epoch-elapsed 0h10m31s total-elapsed 0h10m31s
epoch 1 batch 195/1,245 processed 6,240 batch-loss 15.6784 grad-norm 110.0350 epoch-elapsed 0h10m35s total-elapsed 0h10m35s
epoch 1 batch 196/1,245 processed 6,272 batch-loss 16.8477 grad-norm 262.1567 epoch-elapsed 0h10m39s total-elapsed 0h10m39s
epoch 1 batch 197/1,245 processed 6,304 batch-loss 10.8844 grad-norm 111.2887 epoch-elapsed 0h10m42s total-elapsed 0h10m42s
epoch 1 batch 198/1,245 processed 6,336 batch-loss 10.4330 grad-norm 126.5529 epoch-elapsed 0h10m46s total-elapsed 0h10m46s
epoch 1 batch 199/1,245 processed 6,368 batch-loss 11.2165 grad-norm 96.2823 epoch-elapsed 0h10m50s total-elapsed 0h10m50s
epoch 1 batch 200/1,245 processed 6,400 batch-loss 10.8047 grad-norm 115.6991 epoch-elapsed 0h10m53s total-elapsed 0h10m53s
epoch 1 batch 201/1,245 processed 6,432 batch-loss 16.1658 grad-norm 145.1350 epoch-elapsed 0h10m58s total-elapsed 0h10m58s
epoch 1 batch 202/1,245 processed 6,464 batch-loss 14.7297 grad-norm 149.6821 epoch-elapsed 0h11m01s total-elapsed 0h11m02s
epoch 1 batch 203/1,245 processed 6,496 batch-loss 11.2416 grad-norm 118.1322 epoch-elapsed 0h11m06s total-elapsed 0h11m06s
epoch 1 batch 204/1,245 processed 6,528 batch-loss 13.4325 grad-norm 119.6321 epoch-elapsed 0h11m09s total-elapsed 0h11m09s
epoch 1 batch 205/1,245 processed 6,560 batch-loss 15.2061 grad-norm 116.0162 epoch-elapsed 0h11m13s total-elapsed 0h11m13s
epoch 1 batch 206/1,245 processed 6,592 batch-loss 10.6709 grad-norm 100.3627 epoch-elapsed 0h11m16s total-elapsed 0h11m16s
epoch 1 batch 207/1,245 processed 6,624 batch-loss 13.9605 grad-norm 216.9796 epoch-elapsed 0h11m21s total-elapsed 0h11m21s
epoch 1 batch 208/1,245 processed 6,656 batch-loss 12.3381 grad-norm 85.2398 epoch-elapsed 0h11m24s total-elapsed 0h11m24s
epoch 1 batch 209/1,245 processed 6,688 batch-loss 11.3105 grad-norm 97.2337 epoch-elapsed 0h11m27s total-elapsed 0h11m27s
epoch 1 batch 210/1,245 processed 6,720 batch-loss 13.8772 grad-norm 213.9404 epoch-elapsed 0h11m30s total-elapsed 0h11m30s
epoch 1 batch 211/1,245 processed 6,752 batch-loss 12.8616 grad-norm 174.9807 epoch-elapsed 0h11m33s total-elapsed 0h11m33s
epoch 1 batch 212/1,245 processed 6,784 batch-loss 12.8947 grad-norm 209.4128 epoch-elapsed 0h11m36s total-elapsed 0h11m36s
epoch 1 batch 213/1,245 processed 6,816 batch-loss 12.3306 grad-norm 101.0725 epoch-elapsed 0h11m40s total-elapsed 0h11m40s
epoch 1 batch 214/1,245 processed 6,848 batch-loss 10.9197 grad-norm 76.5257 epoch-elapsed 0h11m42s total-elapsed 0h11m42s
epoch 1 batch 215/1,245 processed 6,880 batch-loss 8.9294 grad-norm 75.9013 epoch-elapsed 0h11m45s total-elapsed 0h11m45s
epoch 1 batch 216/1,245 processed 6,912 batch-loss 9.5487 grad-norm 209.7899 epoch-elapsed 0h11m48s total-elapsed 0h11m48s
epoch 1 batch 217/1,245 processed 6,944 batch-loss 9.4011 grad-norm 189.6067 epoch-elapsed 0h11m50s total-elapsed 0h11m50s
epoch 1 batch 218/1,245 processed 6,976 batch-loss 12.0424 grad-norm 113.3905 epoch-elapsed 0h11m54s total-elapsed 0h11m54s
epoch 1 batch 219/1,245 processed 7,008 batch-loss 10.5658 grad-norm 151.3955 epoch-elapsed 0h11m56s total-elapsed 0h11m56s
epoch 1 batch 220/1,245 processed 7,040 batch-loss 12.8829 grad-norm 89.0341 epoch-elapsed 0h11m59s total-elapsed 0h11m59s
epoch 1 batch 221/1,245 processed 7,072 batch-loss 10.1007 grad-norm 74.5674 epoch-elapsed 0h12m03s total-elapsed 0h12m03s
epoch 1 batch 222/1,245 processed 7,104 batch-loss 12.7403 grad-norm 109.3944 epoch-elapsed 0h12m06s total-elapsed 0h12m06s
epoch 1 batch 223/1,245 processed 7,136 batch-loss 11.2158 grad-norm 271.5021 epoch-elapsed 0h12m08s total-elapsed 0h12m08s
epoch 1 batch 224/1,245 processed 7,168 batch-loss 7.8165 grad-norm 151.7626 epoch-elapsed 0h12m10s total-elapsed 0h12m10s
epoch 1 batch 225/1,245 processed 7,200 batch-loss 8.4804 grad-norm 59.5613 epoch-elapsed 0h12m12s total-elapsed 0h12m12s
epoch 1 batch 226/1,245 processed 7,232 batch-loss 9.5604 grad-norm 122.7798 epoch-elapsed 0h12m15s total-elapsed 0h12m15s
epoch 1 batch 227/1,245 processed 7,264 batch-loss 16.1649 grad-norm 103.8075 epoch-elapsed 0h12m19s total-elapsed 0h12m19s
epoch 1 batch 228/1,245 processed 7,296 batch-loss 9.7706 grad-norm 220.7998 epoch-elapsed 0h12m22s total-elapsed 0h12m22s
epoch 1 batch 229/1,245 processed 7,328 batch-loss 14.1446 grad-norm 156.0674 epoch-elapsed 0h12m25s total-elapsed 0h12m25s
epoch 1 batch 230/1,245 processed 7,360 batch-loss 9.9547 grad-norm 120.4183 epoch-elapsed 0h12m27s total-elapsed 0h12m27s
epoch 1 batch 231/1,245 processed 7,392 batch-loss 13.1210 grad-norm 430.4872 epoch-elapsed 0h12m30s total-elapsed 0h12m30s
epoch 1 batch 232/1,245 processed 7,424 batch-loss 11.4916 grad-norm 104.9747 epoch-elapsed 0h12m33s total-elapsed 0h12m33s
epoch 1 batch 233/1,245 processed 7,456 batch-loss 12.0131 grad-norm 317.2619 epoch-elapsed 0h12m36s total-elapsed 0h12m36s
epoch 1 batch 234/1,245 processed 7,488 batch-loss 9.7582 grad-norm 89.2605 epoch-elapsed 0h12m39s total-elapsed 0h12m39s
epoch 1 batch 235/1,245 processed 7,520 batch-loss 12.8140 grad-norm 333.3319 epoch-elapsed 0h12m43s total-elapsed 0h12m43s
epoch 1 batch 236/1,245 processed 7,552 batch-loss 8.7088 grad-norm 79.8046 epoch-elapsed 0h12m45s total-elapsed 0h12m45s
epoch 1 batch 237/1,245 processed 7,584 batch-loss 11.0420 grad-norm 126.2606 epoch-elapsed 0h12m48s total-elapsed 0h12m48s
epoch 1 batch 238/1,245 processed 7,616 batch-loss 8.5964 grad-norm 969.7242 epoch-elapsed 0h12m51s total-elapsed 0h12m51s
epoch 1 batch 239/1,245 processed 7,648 batch-loss 8.8577 grad-norm 187.3827 epoch-elapsed 0h12m53s total-elapsed 0h12m53s
epoch 1 batch 240/1,245 processed 7,680 batch-loss 10.0829 grad-norm 173.6609 epoch-elapsed 0h12m56s total-elapsed 0h12m56s
epoch 1 batch 241/1,245 processed 7,712 batch-loss 6.3684 grad-norm 66.3147 epoch-elapsed 0h12m59s total-elapsed 0h12m59s
epoch 1 batch 242/1,245 processed 7,744 batch-loss 8.8625 grad-norm 91.4039 epoch-elapsed 0h13m01s total-elapsed 0h13m01s
epoch 1 batch 243/1,245 processed 7,776 batch-loss 14.8881 grad-norm 134.1433 epoch-elapsed 0h13m05s total-elapsed 0h13m05s
epoch 1 batch 244/1,245 processed 7,808 batch-loss 10.5528 grad-norm 65.6735 epoch-elapsed 0h13m07s total-elapsed 0h13m07s
epoch 1 batch 245/1,245 processed 7,840 batch-loss 15.7975 grad-norm 264.0210 epoch-elapsed 0h13m12s total-elapsed 0h13m12s
epoch 1 batch 246/1,245 processed 7,872 batch-loss 10.7214 grad-norm 165.3185 epoch-elapsed 0h13m15s total-elapsed 0h13m15s
epoch 1 batch 247/1,245 processed 7,904 batch-loss 9.8142 grad-norm 100.5254 epoch-elapsed 0h13m18s total-elapsed 0h13m18s
epoch 1 batch 248/1,245 processed 7,936 batch-loss 9.9912 grad-norm 95.6125 epoch-elapsed 0h13m22s total-elapsed 0h13m22s
epoch 1 batch 249/1,245 processed 7,968 batch-loss 12.5920 grad-norm 209.3176 epoch-elapsed 0h13m26s total-elapsed 0h13m26s
epoch 1 batch 250/1,245 processed 8,000 batch-loss 17.1860 grad-norm 141.4624 epoch-elapsed 0h13m31s total-elapsed 0h13m31s
epoch 1 batch 251/1,245 processed 8,032 batch-loss 10.4452 grad-norm 108.8092 epoch-elapsed 0h13m34s total-elapsed 0h13m34s
epoch 1 batch 252/1,245 processed 8,064 batch-loss 9.6910 grad-norm 64.6406 epoch-elapsed 0h13m37s total-elapsed 0h13m37s
epoch 1 batch 253/1,245 processed 8,096 batch-loss 10.3931 grad-norm 337.7312 epoch-elapsed 0h13m40s total-elapsed 0h13m40s
epoch 1 batch 254/1,245 processed 8,128 batch-loss 14.4359 grad-norm 195.3413 epoch-elapsed 0h13m44s total-elapsed 0h13m44s
epoch 1 batch 255/1,245 processed 8,160 batch-loss 8.0029 grad-norm 93.2515 epoch-elapsed 0h13m47s total-elapsed 0h13m47s
epoch 1 batch 256/1,245 processed 8,192 batch-loss 10.7544 grad-norm 83.7353 epoch-elapsed 0h13m49s total-elapsed 0h13m49s
epoch 1 batch 257/1,245 processed 8,224 batch-loss 9.0730 grad-norm 177.0156 epoch-elapsed 0h13m53s total-elapsed 0h13m53s
epoch 1 batch 258/1,245 processed 8,256 batch-loss 10.8536 grad-norm 116.2450 epoch-elapsed 0h13m56s total-elapsed 0h13m56s
epoch 1 batch 259/1,245 processed 8,288 batch-loss 9.7505 grad-norm 166.1818 epoch-elapsed 0h13m58s total-elapsed 0h13m58s
epoch 1 batch 260/1,245 processed 8,320 batch-loss 10.5038 grad-norm 162.2165 epoch-elapsed 0h14m01s total-elapsed 0h14m01s
epoch 1 batch 261/1,245 processed 8,352 batch-loss 10.5735 grad-norm 254.4734 epoch-elapsed 0h14m04s total-elapsed 0h14m04s
epoch 1 batch 262/1,245 processed 8,384 batch-loss 12.1286 grad-norm 186.9941 epoch-elapsed 0h14m07s total-elapsed 0h14m07s
epoch 1 batch 263/1,245 processed 8,416 batch-loss 10.7460 grad-norm 312.2777 epoch-elapsed 0h14m11s total-elapsed 0h14m11s
epoch 1 batch 264/1,245 processed 8,448 batch-loss 12.2576 grad-norm 179.4503 epoch-elapsed 0h14m14s total-elapsed 0h14m14s
epoch 1 batch 265/1,245 processed 8,480 batch-loss 13.1980 grad-norm 126.1677 epoch-elapsed 0h14m18s total-elapsed 0h14m18s
epoch 1 batch 266/1,245 processed 8,512 batch-loss 10.3746 grad-norm 104.2215 epoch-elapsed 0h14m22s total-elapsed 0h14m22s
epoch 1 batch 267/1,245 processed 8,544 batch-loss 12.2558 grad-norm 100.8487 epoch-elapsed 0h14m25s total-elapsed 0h14m25s
epoch 1 batch 268/1,245 processed 8,576 batch-loss 9.3438 grad-norm 141.3959 epoch-elapsed 0h14m28s total-elapsed 0h14m28s
epoch 1 batch 269/1,245 processed 8,608 batch-loss 9.9496 grad-norm 103.3401 epoch-elapsed 0h14m31s total-elapsed 0h14m31s
epoch 1 batch 270/1,245 processed 8,640 batch-loss 10.7011 grad-norm 190.5392 epoch-elapsed 0h14m33s total-elapsed 0h14m33s
epoch 1 batch 271/1,245 processed 8,672 batch-loss 9.2489 grad-norm 136.6521 epoch-elapsed 0h14m36s total-elapsed 0h14m36s
epoch 1 batch 272/1,245 processed 8,704 batch-loss 10.9610 grad-norm 99.3591 epoch-elapsed 0h14m38s total-elapsed 0h14m38s
epoch 1 batch 273/1,245 processed 8,736 batch-loss 12.2802 grad-norm 110.0736 epoch-elapsed 0h14m41s total-elapsed 0h14m41s
epoch 1 batch 274/1,245 processed 8,768 batch-loss 6.7666 grad-norm 120.8478 epoch-elapsed 0h14m43s total-elapsed 0h14m43s
epoch 1 batch 275/1,245 processed 8,800 batch-loss 9.9249 grad-norm 393.3622 epoch-elapsed 0h14m45s total-elapsed 0h14m45s
epoch 1 batch 276/1,245 processed 8,832 batch-loss 9.9267 grad-norm 125.5424 epoch-elapsed 0h14m48s total-elapsed 0h14m48s
epoch 1 batch 277/1,245 processed 8,864 batch-loss 13.6367 grad-norm 362.3106 epoch-elapsed 0h14m51s total-elapsed 0h14m51s
epoch 1 batch 278/1,245 processed 8,896 batch-loss 14.2772 grad-norm 279.0806 epoch-elapsed 0h14m54s total-elapsed 0h14m54s
epoch 1 batch 279/1,245 processed 8,928 batch-loss 8.6452 grad-norm 799.8395 epoch-elapsed 0h14m56s total-elapsed 0h14m56s
epoch 1 batch 280/1,245 processed 8,960 batch-loss 11.1237 grad-norm 178.9687 epoch-elapsed 0h14m59s total-elapsed 0h14m59s
epoch 1 batch 281/1,245 processed 8,992 batch-loss 8.2040 grad-norm 79.7007 epoch-elapsed 0h15m02s total-elapsed 0h15m02s
epoch 1 batch 282/1,245 processed 9,024 batch-loss 9.2141 grad-norm 127.5333 epoch-elapsed 0h15m05s total-elapsed 0h15m05s
epoch 1 batch 283/1,245 processed 9,056 batch-loss 11.8535 grad-norm 331.6407 epoch-elapsed 0h15m09s total-elapsed 0h15m09s
epoch 1 batch 284/1,245 processed 9,088 batch-loss 10.7356 grad-norm 143.6309 epoch-elapsed 0h15m12s total-elapsed 0h15m12s
epoch 1 batch 285/1,245 processed 9,120 batch-loss 10.0585 grad-norm 148.7049 epoch-elapsed 0h15m15s total-elapsed 0h15m15s
epoch 1 batch 286/1,245 processed 9,152 batch-loss 10.6468 grad-norm 279.3041 epoch-elapsed 0h15m18s total-elapsed 0h15m18s
epoch 1 batch 287/1,245 processed 9,184 batch-loss 11.2365 grad-norm 289.8216 epoch-elapsed 0h15m21s total-elapsed 0h15m21s
epoch 1 batch 288/1,245 processed 9,216 batch-loss 11.1614 grad-norm 130.1882 epoch-elapsed 0h15m24s total-elapsed 0h15m24s
epoch 1 batch 289/1,245 processed 9,248 batch-loss 9.8829 grad-norm 359.5512 epoch-elapsed 0h15m27s total-elapsed 0h15m27s
epoch 1 batch 290/1,245 processed 9,280 batch-loss 9.6546 grad-norm 104.8790 epoch-elapsed 0h15m30s total-elapsed 0h15m30s
epoch 1 batch 291/1,245 processed 9,312 batch-loss 13.3716 grad-norm 144.2800 epoch-elapsed 0h15m33s total-elapsed 0h15m33s
epoch 1 batch 292/1,245 processed 9,344 batch-loss 11.5822 grad-norm 161.7654 epoch-elapsed 0h15m36s total-elapsed 0h15m36s
epoch 1 batch 293/1,245 processed 9,376 batch-loss 11.2994 grad-norm 239.9724 epoch-elapsed 0h15m39s total-elapsed 0h15m39s
epoch 1 batch 294/1,245 processed 9,408 batch-loss 11.9247 grad-norm 144.6718 epoch-elapsed 0h15m44s total-elapsed 0h15m44s
epoch 1 batch 295/1,245 processed 9,440 batch-loss 8.7783 grad-norm 273.7413 epoch-elapsed 0h15m47s total-elapsed 0h15m47s
epoch 1 batch 296/1,245 processed 9,472 batch-loss 6.9911 grad-norm 95.9711 epoch-elapsed 0h15m50s total-elapsed 0h15m50s
epoch 1 batch 297/1,245 processed 9,504 batch-loss 9.9174 grad-norm 81.8969 epoch-elapsed 0h15m52s total-elapsed 0h15m52s
epoch 1 batch 298/1,245 processed 9,536 batch-loss 8.3654 grad-norm 74.9454 epoch-elapsed 0h15m54s total-elapsed 0h15m54s
epoch 1 batch 299/1,245 processed 9,568 batch-loss 11.7102 grad-norm 176.5518 epoch-elapsed 0h15m58s total-elapsed 0h15m58s
epoch 1 batch 300/1,245 processed 9,600 batch-loss 10.7660 grad-norm 129.3539 epoch-elapsed 0h16m01s total-elapsed 0h16m01s
epoch 1 batch 301/1,245 processed 9,632 batch-loss 10.2130 grad-norm 150.4446 epoch-elapsed 0h16m04s total-elapsed 0h16m04s
epoch 1 batch 302/1,245 processed 9,664 batch-loss 10.1848 grad-norm 108.5891 epoch-elapsed 0h16m07s total-elapsed 0h16m07s
epoch 1 batch 303/1,245 processed 9,696 batch-loss 11.1498 grad-norm 166.4564 epoch-elapsed 0h16m09s total-elapsed 0h16m09s
epoch 1 batch 304/1,245 processed 9,728 batch-loss 10.0308 grad-norm 172.2777 epoch-elapsed 0h16m12s total-elapsed 0h16m12s
epoch 1 batch 305/1,245 processed 9,760 batch-loss 8.1133 grad-norm 88.4103 epoch-elapsed 0h16m15s total-elapsed 0h16m15s
epoch 1 batch 306/1,245 processed 9,792 batch-loss 9.1244 grad-norm 149.6733 epoch-elapsed 0h16m17s total-elapsed 0h16m17s
epoch 1 batch 307/1,245 processed 9,824 batch-loss 8.5282 grad-norm 136.8658 epoch-elapsed 0h16m21s total-elapsed 0h16m21s
epoch 1 batch 308/1,245 processed 9,856 batch-loss 6.5048 grad-norm 104.9291 epoch-elapsed 0h16m23s total-elapsed 0h16m23s
epoch 1 batch 309/1,245 processed 9,888 batch-loss 9.1825 grad-norm 86.6980 epoch-elapsed 0h16m26s total-elapsed 0h16m26s
epoch 1 batch 310/1,245 processed 9,920 batch-loss 7.5315 grad-norm 110.6274 epoch-elapsed 0h16m28s total-elapsed 0h16m28s
epoch 1 batch 311/1,245 processed 9,952 batch-loss 10.9551 grad-norm 83.7941 epoch-elapsed 0h16m32s total-elapsed 0h16m32s
epoch 1 batch 312/1,245 processed 9,984 batch-loss 11.3946 grad-norm 137.8284 epoch-elapsed 0h16m34s total-elapsed 0h16m34s
dev-fscore (Recall=80.14, Precision=92.16, FScore=85.73, CompleteMatch=15.00) dev-elapsed 0h01m58s total-elapsed 0h18m33s
Saving new best model to models/en_multibert_empty_l2_dev=94.94.pt_dev=85.73...
reindex-elapsed: 0h10m00s
Saving recomputed index
save-elapsed: 0h02m33s
epoch 1 batch 313/1,245 processed 10,016 batch-loss 20.1910 grad-norm 455.4513 epoch-elapsed 0h31m20s total-elapsed 0h31m20s
epoch 1 batch 314/1,245 processed 10,048 batch-loss 13.0781 grad-norm 39.3379 epoch-elapsed 0h31m22s total-elapsed 0h31m22s
epoch 1 batch 315/1,245 processed 10,080 batch-loss 18.4688 grad-norm 62.6264 epoch-elapsed 0h31m26s total-elapsed 0h31m26s
epoch 1 batch 316/1,245 processed 10,112 batch-loss 17.5626 grad-norm 52.3431 epoch-elapsed 0h31m28s total-elapsed 0h31m28s
epoch 1 batch 317/1,245 processed 10,144 batch-loss 17.6702 grad-norm 57.6019 epoch-elapsed 0h31m31s total-elapsed 0h31m31s
epoch 1 batch 318/1,245 processed 10,176 batch-loss 19.9530 grad-norm 114.5722 epoch-elapsed 0h31m35s total-elapsed 0h31m35s
epoch 1 batch 319/1,245 processed 10,208 batch-loss 21.7689 grad-norm 297.8639 epoch-elapsed 0h31m37s total-elapsed 0h31m37s
epoch 1 batch 320/1,245 processed 10,240 batch-loss 21.6870 grad-norm 553.5163 epoch-elapsed 0h31m41s total-elapsed 0h31m41s
epoch 1 batch 321/1,245 processed 10,272 batch-loss 28.7379 grad-norm 119.4826 epoch-elapsed 0h31m44s total-elapsed 0h31m44s
epoch 1 batch 322/1,245 processed 10,304 batch-loss 20.1831 grad-norm 174.6110 epoch-elapsed 0h31m46s total-elapsed 0h31m46s
epoch 1 batch 323/1,245 processed 10,336 batch-loss 28.9651 grad-norm 33.6937 epoch-elapsed 0h31m49s total-elapsed 0h31m49s
epoch 1 batch 324/1,245 processed 10,368 batch-loss 28.4185 grad-norm 507.3681 epoch-elapsed 0h31m51s total-elapsed 0h31m51s
epoch 1 batch 325/1,245 processed 10,400 batch-loss 18.9552 grad-norm 32.6559 epoch-elapsed 0h31m54s total-elapsed 0h31m54s
epoch 1 batch 326/1,245 processed 10,432 batch-loss 35.5553 grad-norm 125.2986 epoch-elapsed 0h31m58s total-elapsed 0h31m58s
epoch 1 batch 327/1,245 processed 10,464 batch-loss 24.7875 grad-norm 34.5500 epoch-elapsed 0h32m01s total-elapsed 0h32m01s
epoch 1 batch 328/1,245 processed 10,496 batch-loss 24.0579 grad-norm 37.4647 epoch-elapsed 0h32m04s total-elapsed 0h32m04s
epoch 1 batch 329/1,245 processed 10,528 batch-loss 20.5693 grad-norm 47.2982 epoch-elapsed 0h32m06s total-elapsed 0h32m06s
epoch 1 batch 330/1,245 processed 10,560 batch-loss 24.6513 grad-norm 298.9921 epoch-elapsed 0h32m09s total-elapsed 0h32m09s
epoch 1 batch 331/1,245 processed 10,592 batch-loss 18.1782 grad-norm 101.9809 epoch-elapsed 0h32m12s total-elapsed 0h32m12s
epoch 1 batch 332/1,245 processed 10,624 batch-loss 14.5891 grad-norm 40.1127 epoch-elapsed 0h32m15s total-elapsed 0h32m15s
epoch 1 batch 333/1,245 processed 10,656 batch-loss 16.9720 grad-norm 153.8087 epoch-elapsed 0h32m17s total-elapsed 0h32m17s
epoch 1 batch 334/1,245 processed 10,688 batch-loss 22.5964 grad-norm 65.3095 epoch-elapsed 0h32m21s total-elapsed 0h32m21s
epoch 1 batch 335/1,245 processed 10,720 batch-loss 19.9501 grad-norm 215.8202 epoch-elapsed 0h32m23s total-elapsed 0h32m23s
epoch 1 batch 336/1,245 processed 10,752 batch-loss 14.0996 grad-norm 30.3832 epoch-elapsed 0h32m26s total-elapsed 0h32m26s
epoch 1 batch 337/1,245 processed 10,784 batch-loss 26.8790 grad-norm 49.4730 epoch-elapsed 0h32m29s total-elapsed 0h32m29s
epoch 1 batch 338/1,245 processed 10,816 batch-loss 16.1449 grad-norm 227.5406 epoch-elapsed 0h32m32s total-elapsed 0h32m32s
epoch 1 batch 339/1,245 processed 10,848 batch-loss 15.5026 grad-norm 47.1229 epoch-elapsed 0h32m34s total-elapsed 0h32m34s
epoch 1 batch 340/1,245 processed 10,880 batch-loss 20.8606 grad-norm 128.3574 epoch-elapsed 0h32m37s total-elapsed 0h32m37s
epoch 1 batch 341/1,245 processed 10,912 batch-loss 29.3315 grad-norm 161.9290 epoch-elapsed 0h32m41s total-elapsed 0h32m41s
epoch 1 batch 342/1,245 processed 10,944 batch-loss 17.2812 grad-norm 123.6579 epoch-elapsed 0h32m43s total-elapsed 0h32m43s
epoch 1 batch 343/1,245 processed 10,976 batch-loss 15.2824 grad-norm 33.7681 epoch-elapsed 0h32m45s total-elapsed 0h32m45s
epoch 1 batch 344/1,245 processed 11,008 batch-loss 20.8710 grad-norm 294.0399 epoch-elapsed 0h32m48s total-elapsed 0h32m48s
epoch 1 batch 345/1,245 processed 11,040 batch-loss 21.3420 grad-norm 412.1226 epoch-elapsed 0h32m50s total-elapsed 0h32m50s
epoch 1 batch 346/1,245 processed 11,072 batch-loss 22.9645 grad-norm 43.5987 epoch-elapsed 0h32m53s total-elapsed 0h32m53s
epoch 1 batch 347/1,245 processed 11,104 batch-loss 18.7414 grad-norm 50.6489 epoch-elapsed 0h32m56s total-elapsed 0h32m56s
epoch 1 batch 348/1,245 processed 11,136 batch-loss 16.2628 grad-norm 112.8465 epoch-elapsed 0h32m58s total-elapsed 0h32m58s
epoch 1 batch 349/1,245 processed 11,168 batch-loss 22.4513 grad-norm 197.0615 epoch-elapsed 0h33m02s total-elapsed 0h33m02s
epoch 1 batch 350/1,245 processed 11,200 batch-loss 23.1391 grad-norm 231.3896 epoch-elapsed 0h33m05s total-elapsed 0h33m05s
epoch 1 batch 351/1,245 processed 11,232 batch-loss 20.1158 grad-norm 222.8028 epoch-elapsed 0h33m08s total-elapsed 0h33m08s
epoch 1 batch 352/1,245 processed 11,264 batch-loss 20.2261 grad-norm 132.0565 epoch-elapsed 0h33m12s total-elapsed 0h33m12s
epoch 1 batch 353/1,245 processed 11,296 batch-loss 21.4164 grad-norm 99.6040 epoch-elapsed 0h33m15s total-elapsed 0h33m15s
epoch 1 batch 354/1,245 processed 11,328 batch-loss 22.0421 grad-norm 32.4220 epoch-elapsed 0h33m18s total-elapsed 0h33m18s
epoch 1 batch 355/1,245 processed 11,360 batch-loss 18.4795 grad-norm 40.2446 epoch-elapsed 0h33m21s total-elapsed 0h33m21s
epoch 1 batch 356/1,245 processed 11,392 batch-loss 15.8911 grad-norm 45.3429 epoch-elapsed 0h33m24s total-elapsed 0h33m24s
epoch 1 batch 357/1,245 processed 11,424 batch-loss 11.9080 grad-norm 106.4760 epoch-elapsed 0h33m26s total-elapsed 0h33m26s
epoch 1 batch 358/1,245 processed 11,456 batch-loss 14.3199 grad-norm 200.7635 epoch-elapsed 0h33m28s total-elapsed 0h33m28s
epoch 1 batch 359/1,245 processed 11,488 batch-loss 19.3459 grad-norm 319.1819 epoch-elapsed 0h33m31s total-elapsed 0h33m31s
epoch 1 batch 360/1,245 processed 11,520 batch-loss 18.4712 grad-norm 63.9314 epoch-elapsed 0h33m33s total-elapsed 0h33m33s
epoch 1 batch 361/1,245 processed 11,552 batch-loss 12.4562 grad-norm 133.2143 epoch-elapsed 0h33m36s total-elapsed 0h33m36s
epoch 1 batch 362/1,245 processed 11,584 batch-loss 18.5119 grad-norm 101.3783 epoch-elapsed 0h33m39s total-elapsed 0h33m39s
epoch 1 batch 363/1,245 processed 11,616 batch-loss 22.9269 grad-norm 198.9517 epoch-elapsed 0h33m42s total-elapsed 0h33m42s
epoch 1 batch 364/1,245 processed 11,648 batch-loss 13.9425 grad-norm 174.2089 epoch-elapsed 0h33m45s total-elapsed 0h33m45s
epoch 1 batch 365/1,245 processed 11,680 batch-loss 14.4640 grad-norm 48.2380 epoch-elapsed 0h33m48s total-elapsed 0h33m48s
epoch 1 batch 366/1,245 processed 11,712 batch-loss 18.1843 grad-norm 92.7117 epoch-elapsed 0h33m51s total-elapsed 0h33m51s
epoch 1 batch 367/1,245 processed 11,744 batch-loss 19.5458 grad-norm 108.8851 epoch-elapsed 0h33m54s total-elapsed 0h33m54s
epoch 1 batch 368/1,245 processed 11,776 batch-loss 20.5481 grad-norm 129.1165 epoch-elapsed 0h33m58s total-elapsed 0h33m58s
epoch 1 batch 369/1,245 processed 11,808 batch-loss 17.0796 grad-norm 121.7327 epoch-elapsed 0h34m01s total-elapsed 0h34m01s
epoch 1 batch 370/1,245 processed 11,840 batch-loss 16.0982 grad-norm 67.7408 epoch-elapsed 0h34m03s total-elapsed 0h34m03s
epoch 1 batch 371/1,245 processed 11,872 batch-loss 19.0142 grad-norm 180.7549 epoch-elapsed 0h34m06s total-elapsed 0h34m06s
epoch 1 batch 372/1,245 processed 11,904 batch-loss 13.7800 grad-norm 121.2113 epoch-elapsed 0h34m08s total-elapsed 0h34m08s
epoch 1 batch 373/1,245 processed 11,936 batch-loss 15.2082 grad-norm 583.6523 epoch-elapsed 0h34m11s total-elapsed 0h34m11s
epoch 1 batch 374/1,245 processed 11,968 batch-loss 14.5524 grad-norm 170.4395 epoch-elapsed 0h34m14s total-elapsed 0h34m14s
epoch 1 batch 375/1,245 processed 12,000 batch-loss 17.5089 grad-norm 618.3662 epoch-elapsed 0h34m17s total-elapsed 0h34m17s
epoch 1 batch 376/1,245 processed 12,032 batch-loss 13.7543 grad-norm 305.8895 epoch-elapsed 0h34m19s total-elapsed 0h34m19s
epoch 1 batch 377/1,245 processed 12,064 batch-loss 12.7664 grad-norm 108.0302 epoch-elapsed 0h34m21s total-elapsed 0h34m21s
epoch 1 batch 378/1,245 processed 12,096 batch-loss 12.7953 grad-norm 214.5417 epoch-elapsed 0h34m24s total-elapsed 0h34m24s
epoch 1 batch 379/1,245 processed 12,128 batch-loss 15.2170 grad-norm 178.9094 epoch-elapsed 0h34m26s total-elapsed 0h34m26s
epoch 1 batch 380/1,245 processed 12,160 batch-loss 12.3953 grad-norm 289.6226 epoch-elapsed 0h34m29s total-elapsed 0h34m29s
epoch 1 batch 381/1,245 processed 12,192 batch-loss 15.1968 grad-norm 300.0574 epoch-elapsed 0h34m32s total-elapsed 0h34m32s
epoch 1 batch 382/1,245 processed 12,224 batch-loss 12.4906 grad-norm 278.8888 epoch-elapsed 0h34m35s total-elapsed 0h34m35s
epoch 1 batch 383/1,245 processed 12,256 batch-loss 13.8614 grad-norm 321.6232 epoch-elapsed 0h34m38s total-elapsed 0h34m38s
epoch 1 batch 384/1,245 processed 12,288 batch-loss 12.5655 grad-norm 42.7225 epoch-elapsed 0h34m40s total-elapsed 0h34m40s
epoch 1 batch 385/1,245 processed 12,320 batch-loss 14.9130 grad-norm 40.0661 epoch-elapsed 0h34m43s total-elapsed 0h34m43s
epoch 1 batch 386/1,245 processed 12,352 batch-loss 13.0711 grad-norm 191.6740 epoch-elapsed 0h34m45s total-elapsed 0h34m45s
epoch 1 batch 387/1,245 processed 12,384 batch-loss 15.6733 grad-norm 148.2271 epoch-elapsed 0h34m48s total-elapsed 0h34m48s
epoch 1 batch 388/1,245 processed 12,416 batch-loss 11.9679 grad-norm 82.8529 epoch-elapsed 0h34m51s total-elapsed 0h34m51s
epoch 1 batch 389/1,245 processed 12,448 batch-loss 16.4815 grad-norm 522.5218 epoch-elapsed 0h34m58s total-elapsed 0h34m58s
epoch 1 batch 390/1,245 processed 12,480 batch-loss 10.8582 grad-norm 114.1155 epoch-elapsed 0h35m00s total-elapsed 0h35m00s
epoch 1 batch 391/1,245 processed 12,512 batch-loss 10.9605 grad-norm 41.4356 epoch-elapsed 0h35m02s total-elapsed 0h35m02s
epoch 1 batch 392/1,245 processed 12,544 batch-loss 14.5020 grad-norm 66.7201 epoch-elapsed 0h35m05s total-elapsed 0h35m05s
epoch 1 batch 393/1,245 processed 12,576 batch-loss 15.0801 grad-norm 141.0011 epoch-elapsed 0h35m08s total-elapsed 0h35m08s
epoch 1 batch 394/1,245 processed 12,608 batch-loss 12.8071 grad-norm 569.1596 epoch-elapsed 0h35m10s total-elapsed 0h35m10s
epoch 1 batch 395/1,245 processed 12,640 batch-loss 11.9056 grad-norm 126.0672 epoch-elapsed 0h35m14s total-elapsed 0h35m14s
epoch 1 batch 396/1,245 processed 12,672 batch-loss 11.9296 grad-norm 112.7225 epoch-elapsed 0h35m17s total-elapsed 0h35m17s
epoch 1 batch 397/1,245 processed 12,704 batch-loss 12.8902 grad-norm 779.9498 epoch-elapsed 0h35m20s total-elapsed 0h35m20s
epoch 1 batch 398/1,245 processed 12,736 batch-loss 9.8665 grad-norm 107.9717 epoch-elapsed 0h35m22s total-elapsed 0h35m22s
epoch 1 batch 399/1,245 processed 12,768 batch-loss 10.3027 grad-norm 167.2510 epoch-elapsed 0h35m25s total-elapsed 0h35m25s
epoch 1 batch 400/1,245 processed 12,800 batch-loss 18.2216 grad-norm 293.0695 epoch-elapsed 0h35m28s total-elapsed 0h35m28s
epoch 1 batch 401/1,245 processed 12,832 batch-loss 12.4044 grad-norm 52.9322 epoch-elapsed 0h35m31s total-elapsed 0h35m31s
epoch 1 batch 402/1,245 processed 12,864 batch-loss 13.1248 grad-norm 779.6255 epoch-elapsed 0h35m35s total-elapsed 0h35m35s
epoch 1 batch 403/1,245 processed 12,896 batch-loss 9.1296 grad-norm 35.3933 epoch-elapsed 0h35m38s total-elapsed 0h35m38s
epoch 1 batch 404/1,245 processed 12,928 batch-loss 12.7137 grad-norm 218.1927 epoch-elapsed 0h35m40s total-elapsed 0h35m40s
epoch 1 batch 405/1,245 processed 12,960 batch-loss 11.1577 grad-norm 72.2251 epoch-elapsed 0h35m43s total-elapsed 0h35m43s
epoch 1 batch 406/1,245 processed 12,992 batch-loss 11.4684 grad-norm 86.8454 epoch-elapsed 0h35m46s total-elapsed 0h35m46s
epoch 1 batch 407/1,245 processed 13,024 batch-loss 12.0787 grad-norm 58.6004 epoch-elapsed 0h35m48s total-elapsed 0h35m48s
epoch 1 batch 408/1,245 processed 13,056 batch-loss 11.3734 grad-norm 114.1678 epoch-elapsed 0h35m51s total-elapsed 0h35m51s
epoch 1 batch 409/1,245 processed 13,088 batch-loss 10.3124 grad-norm 120.5481 epoch-elapsed 0h35m54s total-elapsed 0h35m54s
epoch 1 batch 410/1,245 processed 13,120 batch-loss 11.4970 grad-norm 278.4545 epoch-elapsed 0h35m57s total-elapsed 0h35m57s
epoch 1 batch 411/1,245 processed 13,152 batch-loss 10.8768 grad-norm 83.1406 epoch-elapsed 0h36m00s total-elapsed 0h36m00s
epoch 1 batch 412/1,245 processed 13,184 batch-loss 8.4991 grad-norm 47.8910 epoch-elapsed 0h36m01s total-elapsed 0h36m01s
epoch 1 batch 413/1,245 processed 13,216 batch-loss 8.9507 grad-norm 74.1432 epoch-elapsed 0h36m04s total-elapsed 0h36m04s
epoch 1 batch 414/1,245 processed 13,248 batch-loss 6.6528 grad-norm 41.4447 epoch-elapsed 0h36m06s total-elapsed 0h36m06s
epoch 1 batch 415/1,245 processed 13,280 batch-loss 8.3860 grad-norm 31.7962 epoch-elapsed 0h36m08s total-elapsed 0h36m08s
epoch 1 batch 416/1,245 processed 13,312 batch-loss 18.8149 grad-norm 149.2727 epoch-elapsed 0h36m12s total-elapsed 0h36m12s
epoch 1 batch 417/1,245 processed 13,344 batch-loss 10.7946 grad-norm 200.0017 epoch-elapsed 0h36m15s total-elapsed 0h36m15s
epoch 1 batch 418/1,245 processed 13,376 batch-loss 12.2825 grad-norm 98.0460 epoch-elapsed 0h36m19s total-elapsed 0h36m19s
epoch 1 batch 419/1,245 processed 13,408 batch-loss 8.6329 grad-norm 76.1273 epoch-elapsed 0h36m22s total-elapsed 0h36m22s
epoch 1 batch 420/1,245 processed 13,440 batch-loss 8.3541 grad-norm 48.8532 epoch-elapsed 0h36m25s total-elapsed 0h36m25s
epoch 1 batch 421/1,245 processed 13,472 batch-loss 9.5993 grad-norm 87.4412 epoch-elapsed 0h36m28s total-elapsed 0h36m28s
epoch 1 batch 422/1,245 processed 13,504 batch-loss 9.5835 grad-norm 125.6461 epoch-elapsed 0h36m31s total-elapsed 0h36m31s
epoch 1 batch 423/1,245 processed 13,536 batch-loss 8.6477 grad-norm 45.3405 epoch-elapsed 0h36m33s total-elapsed 0h36m33s
epoch 1 batch 424/1,245 processed 13,568 batch-loss 10.6129 grad-norm 192.6138 epoch-elapsed 0h36m36s total-elapsed 0h36m36s
epoch 1 batch 425/1,245 processed 13,600 batch-loss 11.0936 grad-norm 159.7759 epoch-elapsed 0h36m39s total-elapsed 0h36m39s
epoch 1 batch 426/1,245 processed 13,632 batch-loss 9.1442 grad-norm 53.0020 epoch-elapsed 0h36m41s total-elapsed 0h36m41s
epoch 1 batch 427/1,245 processed 13,664 batch-loss 12.0485 grad-norm 185.6493 epoch-elapsed 0h36m44s total-elapsed 0h36m44s
epoch 1 batch 428/1,245 processed 13,696 batch-loss 9.7876 grad-norm 95.3197 epoch-elapsed 0h36m47s total-elapsed 0h36m47s
epoch 1 batch 429/1,245 processed 13,728 batch-loss 7.9256 grad-norm 65.0931 epoch-elapsed 0h36m49s total-elapsed 0h36m49s
epoch 1 batch 430/1,245 processed 13,760 batch-loss 9.4396 grad-norm 132.9012 epoch-elapsed 0h36m52s total-elapsed 0h36m52s
epoch 1 batch 431/1,245 processed 13,792 batch-loss 7.6735 grad-norm 94.1376 epoch-elapsed 0h36m56s total-elapsed 0h36m56s
epoch 1 batch 432/1,245 processed 13,824 batch-loss 6.8532 grad-norm 286.9672 epoch-elapsed 0h36m59s total-elapsed 0h36m59s
epoch 1 batch 433/1,245 processed 13,856 batch-loss 8.4517 grad-norm 74.6143 epoch-elapsed 0h37m01s total-elapsed 0h37m01s
epoch 1 batch 434/1,245 processed 13,888 batch-loss 9.9243 grad-norm 63.5224 epoch-elapsed 0h37m04s total-elapsed 0h37m04s
epoch 1 batch 435/1,245 processed 13,920 batch-loss 7.2685 grad-norm 98.0090 epoch-elapsed 0h37m07s total-elapsed 0h37m07s
epoch 1 batch 436/1,245 processed 13,952 batch-loss 8.8627 grad-norm 46.4342 epoch-elapsed 0h37m09s total-elapsed 0h37m09s
epoch 1 batch 437/1,245 processed 13,984 batch-loss 10.1299 grad-norm 80.9106 epoch-elapsed 0h37m12s total-elapsed 0h37m12s
epoch 1 batch 438/1,245 processed 14,016 batch-loss 8.6108 grad-norm 285.2069 epoch-elapsed 0h37m14s total-elapsed 0h37m14s
epoch 1 batch 439/1,245 processed 14,048 batch-loss 9.6669 grad-norm 80.0202 epoch-elapsed 0h37m17s total-elapsed 0h37m17s
epoch 1 batch 440/1,245 processed 14,080 batch-loss 13.8042 grad-norm 126.1058 epoch-elapsed 0h37m21s total-elapsed 0h37m21s
epoch 1 batch 441/1,245 processed 14,112 batch-loss 8.2654 grad-norm 73.1637 epoch-elapsed 0h37m24s total-elapsed 0h37m24s
epoch 1 batch 442/1,245 processed 14,144 batch-loss 9.2172 grad-norm 79.8404 epoch-elapsed 0h37m27s total-elapsed 0h37m27s
epoch 1 batch 443/1,245 processed 14,176 batch-loss 8.1299 grad-norm 46.4084 epoch-elapsed 0h37m29s total-elapsed 0h37m29s
epoch 1 batch 444/1,245 processed 14,208 batch-loss 7.4324 grad-norm 75.2336 epoch-elapsed 0h37m32s total-elapsed 0h37m32s
epoch 1 batch 445/1,245 processed 14,240 batch-loss 9.3120 grad-norm 67.9751 epoch-elapsed 0h37m36s total-elapsed 0h37m36s
epoch 1 batch 446/1,245 processed 14,272 batch-loss 9.6100 grad-norm 319.4669 epoch-elapsed 0h37m38s total-elapsed 0h37m38s
epoch 1 batch 447/1,245 processed 14,304 batch-loss 7.5393 grad-norm 71.2549 epoch-elapsed 0h37m41s total-elapsed 0h37m41s
epoch 1 batch 448/1,245 processed 14,336 batch-loss 9.7020 grad-norm 111.7986 epoch-elapsed 0h37m44s total-elapsed 0h37m44s
epoch 1 batch 449/1,245 processed 14,368 batch-loss 7.1201 grad-norm 44.3228 epoch-elapsed 0h37m47s total-elapsed 0h37m47s
epoch 1 batch 450/1,245 processed 14,400 batch-loss 8.4056 grad-norm 135.4135 epoch-elapsed 0h37m49s total-elapsed 0h37m49s
epoch 1 batch 451/1,245 processed 14,432 batch-loss 8.7878 grad-norm 147.5191 epoch-elapsed 0h37m53s total-elapsed 0h37m53s
epoch 1 batch 452/1,245 processed 14,464 batch-loss 6.9291 grad-norm 61.1097 epoch-elapsed 0h37m55s total-elapsed 0h37m55s
epoch 1 batch 453/1,245 processed 14,496 batch-loss 7.7265 grad-norm 102.5257 epoch-elapsed 0h37m57s total-elapsed 0h37m57s
epoch 1 batch 454/1,245 processed 14,528 batch-loss 7.9663 grad-norm 1438.1860 epoch-elapsed 0h37m59s total-elapsed 0h37m59s
epoch 1 batch 455/1,245 processed 14,560 batch-loss 12.0705 grad-norm 83.3590 epoch-elapsed 0h38m03s total-elapsed 0h38m03s
epoch 1 batch 456/1,245 processed 14,592 batch-loss 6.3895 grad-norm 253.0131 epoch-elapsed 0h38m06s total-elapsed 0h38m06s
epoch 1 batch 457/1,245 processed 14,624 batch-loss 9.2847 grad-norm 117.1130 epoch-elapsed 0h38m09s total-elapsed 0h38m09s
epoch 1 batch 458/1,245 processed 14,656 batch-loss 10.3695 grad-norm 270.1316 epoch-elapsed 0h38m13s total-elapsed 0h38m13s
epoch 1 batch 459/1,245 processed 14,688 batch-loss 9.3148 grad-norm 95.5158 epoch-elapsed 0h38m16s total-elapsed 0h38m16s
epoch 1 batch 460/1,245 processed 14,720 batch-loss 6.1878 grad-norm 87.5440 epoch-elapsed 0h38m19s total-elapsed 0h38m19s
epoch 1 batch 461/1,245 processed 14,752 batch-loss 8.7484 grad-norm 79.5715 epoch-elapsed 0h38m21s total-elapsed 0h38m21s
epoch 1 batch 462/1,245 processed 14,784 batch-loss 6.9051 grad-norm 60.0783 epoch-elapsed 0h38m23s total-elapsed 0h38m23s
epoch 1 batch 463/1,245 processed 14,816 batch-loss 9.4610 grad-norm 103.1390 epoch-elapsed 0h38m27s total-elapsed 0h38m27s
epoch 1 batch 464/1,245 processed 14,848 batch-loss 7.0691 grad-norm 57.4077 epoch-elapsed 0h38m29s total-elapsed 0h38m29s
epoch 1 batch 465/1,245 processed 14,880 batch-loss 9.0501 grad-norm 194.2989 epoch-elapsed 0h38m32s total-elapsed 0h38m32s
epoch 1 batch 466/1,245 processed 14,912 batch-loss 5.9426 grad-norm 79.2805 epoch-elapsed 0h38m35s total-elapsed 0h38m35s
epoch 1 batch 467/1,245 processed 14,944 batch-loss 8.1458 grad-norm 50.8815 epoch-elapsed 0h38m38s total-elapsed 0h38m38s
epoch 1 batch 468/1,245 processed 14,976 batch-loss 9.1505 grad-norm 473.1135 epoch-elapsed 0h38m41s total-elapsed 0h38m41s
epoch 1 batch 469/1,245 processed 15,008 batch-loss 8.0005 grad-norm 281.5896 epoch-elapsed 0h38m44s total-elapsed 0h38m44s
epoch 1 batch 470/1,245 processed 15,040 batch-loss 7.2189 grad-norm 87.6537 epoch-elapsed 0h38m47s total-elapsed 0h38m47s
epoch 1 batch 471/1,245 processed 15,072 batch-loss 9.9757 grad-norm 214.0887 epoch-elapsed 0h38m50s total-elapsed 0h38m50s
epoch 1 batch 472/1,245 processed 15,104 batch-loss 6.9701 grad-norm 80.1834 epoch-elapsed 0h38m53s total-elapsed 0h38m53s
epoch 1 batch 473/1,245 processed 15,136 batch-loss 9.2559 grad-norm 101.3454 epoch-elapsed 0h38m57s total-elapsed 0h38m57s
epoch 1 batch 474/1,245 processed 15,168 batch-loss 6.3224 grad-norm 50.6197 epoch-elapsed 0h38m59s total-elapsed 0h38m59s
epoch 1 batch 475/1,245 processed 15,200 batch-loss 7.7611 grad-norm 489.7175 epoch-elapsed 0h39m01s total-elapsed 0h39m01s
epoch 1 batch 476/1,245 processed 15,232 batch-loss 8.4898 grad-norm 68.6453 epoch-elapsed 0h39m04s total-elapsed 0h39m04s
epoch 1 batch 477/1,245 processed 15,264 batch-loss 5.7638 grad-norm 45.7053 epoch-elapsed 0h39m07s total-elapsed 0h39m07s
epoch 1 batch 478/1,245 processed 15,296 batch-loss 9.6900 grad-norm 163.1212 epoch-elapsed 0h39m09s total-elapsed 0h39m09s
epoch 1 batch 479/1,245 processed 15,328 batch-loss 6.2762 grad-norm 86.3894 epoch-elapsed 0h39m11s total-elapsed 0h39m11s
epoch 1 batch 480/1,245 processed 15,360 batch-loss 6.2341 grad-norm 58.2319 epoch-elapsed 0h39m15s total-elapsed 0h39m15s
epoch 1 batch 481/1,245 processed 15,392 batch-loss 5.7751 grad-norm 70.9017 epoch-elapsed 0h39m18s total-elapsed 0h39m18s
epoch 1 batch 482/1,245 processed 15,424 batch-loss 7.2361 grad-norm 91.1016 epoch-elapsed 0h39m20s total-elapsed 0h39m20s
epoch 1 batch 483/1,245 processed 15,456 batch-loss 5.4788 grad-norm 134.1513 epoch-elapsed 0h39m23s total-elapsed 0h39m23s
epoch 1 batch 484/1,245 processed 15,488 batch-loss 5.6726 grad-norm 178.8865 epoch-elapsed 0h39m25s total-elapsed 0h39m25s
epoch 1 batch 485/1,245 processed 15,520 batch-loss 5.9192 grad-norm 90.4068 epoch-elapsed 0h39m29s total-elapsed 0h39m29s
epoch 1 batch 486/1,245 processed 15,552 batch-loss 10.0022 grad-norm 138.4447 epoch-elapsed 0h39m33s total-elapsed 0h39m33s
epoch 1 batch 487/1,245 processed 15,584 batch-loss 5.1870 grad-norm 87.7468 epoch-elapsed 0h39m35s total-elapsed 0h39m35s
epoch 1 batch 488/1,245 processed 15,616 batch-loss 8.7081 grad-norm 745.8762 epoch-elapsed 0h39m37s total-elapsed 0h39m37s
epoch 1 batch 489/1,245 processed 15,648 batch-loss 6.4359 grad-norm 57.9229 epoch-elapsed 0h39m41s total-elapsed 0h39m41s
epoch 1 batch 490/1,245 processed 15,680 batch-loss 7.8435 grad-norm 50.0702 epoch-elapsed 0h39m43s total-elapsed 0h39m43s
epoch 1 batch 491/1,245 processed 15,712 batch-loss 8.2272 grad-norm 134.4130 epoch-elapsed 0h39m45s total-elapsed 0h39m45s
epoch 1 batch 492/1,245 processed 15,744 batch-loss 6.9994 grad-norm 153.3900 epoch-elapsed 0h39m49s total-elapsed 0h39m49s
epoch 1 batch 493/1,245 processed 15,776 batch-loss 7.4310 grad-norm 64.8587 epoch-elapsed 0h39m52s total-elapsed 0h39m52s
epoch 1 batch 494/1,245 processed 15,808 batch-loss 7.5020 grad-norm 175.1620 epoch-elapsed 0h39m55s total-elapsed 0h39m55s
epoch 1 batch 495/1,245 processed 15,840 batch-loss 6.1696 grad-norm 47.6871 epoch-elapsed 0h39m58s total-elapsed 0h39m58s
epoch 1 batch 496/1,245 processed 15,872 batch-loss 6.7807 grad-norm 112.3377 epoch-elapsed 0h40m00s total-elapsed 0h40m00s
epoch 1 batch 497/1,245 processed 15,904 batch-loss 8.0743 grad-norm 154.6318 epoch-elapsed 0h40m03s total-elapsed 0h40m03s
epoch 1 batch 498/1,245 processed 15,936 batch-loss 6.4074 grad-norm 77.8247 epoch-elapsed 0h40m06s total-elapsed 0h40m06s
epoch 1 batch 499/1,245 processed 15,968 batch-loss 7.9245 grad-norm 175.7284 epoch-elapsed 0h40m09s total-elapsed 0h40m09s
epoch 1 batch 500/1,245 processed 16,000 batch-loss 3.0253 grad-norm 54.7353 epoch-elapsed 0h40m11s total-elapsed 0h40m11s
epoch 1 batch 501/1,245 processed 16,032 batch-loss 5.8200 grad-norm 82.3365 epoch-elapsed 0h40m14s total-elapsed 0h40m14s
epoch 1 batch 502/1,245 processed 16,064 batch-loss 6.7295 grad-norm 74.3332 epoch-elapsed 0h40m17s total-elapsed 0h40m17s
epoch 1 batch 503/1,245 processed 16,096 batch-loss 9.4080 grad-norm 591.3410 epoch-elapsed 0h40m21s total-elapsed 0h40m21s
epoch 1 batch 504/1,245 processed 16,128 batch-loss 5.4178 grad-norm 162.6764 epoch-elapsed 0h40m24s total-elapsed 0h40m24s
epoch 1 batch 505/1,245 processed 16,160 batch-loss 7.0963 grad-norm 51.4116 epoch-elapsed 0h40m26s total-elapsed 0h40m26s
epoch 1 batch 506/1,245 processed 16,192 batch-loss 7.8346 grad-norm 117.0288 epoch-elapsed 0h40m30s total-elapsed 0h40m30s
epoch 1 batch 507/1,245 processed 16,224 batch-loss 9.6089 grad-norm 158.2167 epoch-elapsed 0h40m34s total-elapsed 0h40m34s
epoch 1 batch 508/1,245 processed 16,256 batch-loss 5.7396 grad-norm 125.2760 epoch-elapsed 0h40m36s total-elapsed 0h40m36s
epoch 1 batch 509/1,245 processed 16,288 batch-loss 9.9882 grad-norm 131.3888 epoch-elapsed 0h40m39s total-elapsed 0h40m39s
epoch 1 batch 510/1,245 processed 16,320 batch-loss 5.8048 grad-norm 105.2331 epoch-elapsed 0h40m41s total-elapsed 0h40m41s
epoch 1 batch 511/1,245 processed 16,352 batch-loss 8.9621 grad-norm 85.6394 epoch-elapsed 0h40m44s total-elapsed 0h40m44s
epoch 1 batch 512/1,245 processed 16,384 batch-loss 9.9067 grad-norm 536.5987 epoch-elapsed 0h40m48s total-elapsed 0h40m48s
epoch 1 batch 513/1,245 processed 16,416 batch-loss 7.1111 grad-norm 100.9272 epoch-elapsed 0h40m50s total-elapsed 0h40m50s
epoch 1 batch 514/1,245 processed 16,448 batch-loss 6.4826 grad-norm 61.2947 epoch-elapsed 0h40m53s total-elapsed 0h40m53s
epoch 1 batch 515/1,245 processed 16,480 batch-loss 2.6566 grad-norm 35.8157 epoch-elapsed 0h40m56s total-elapsed 0h40m56s
epoch 1 batch 516/1,245 processed 16,512 batch-loss 5.5349 grad-norm 217.3282 epoch-elapsed 0h40m59s total-elapsed 0h40m59s
epoch 1 batch 517/1,245 processed 16,544 batch-loss 5.9692 grad-norm 76.0720 epoch-elapsed 0h41m01s total-elapsed 0h41m01s
epoch 1 batch 518/1,245 processed 16,576 batch-loss 5.5328 grad-norm 68.2530 epoch-elapsed 0h41m05s total-elapsed 0h41m05s
epoch 1 batch 519/1,245 processed 16,608 batch-loss 8.5680 grad-norm 108.6126 epoch-elapsed 0h41m09s total-elapsed 0h41m09s
epoch 1 batch 520/1,245 processed 16,640 batch-loss 7.4180 grad-norm 99.0525 epoch-elapsed 0h41m11s total-elapsed 0h41m11s
epoch 1 batch 521/1,245 processed 16,672 batch-loss 6.2916 grad-norm 118.4813 epoch-elapsed 0h41m14s total-elapsed 0h41m14s
epoch 1 batch 522/1,245 processed 16,704 batch-loss 6.2256 grad-norm 175.6000 epoch-elapsed 0h41m17s total-elapsed 0h41m17s
epoch 1 batch 523/1,245 processed 16,736 batch-loss 7.0490 grad-norm 70.1639 epoch-elapsed 0h41m19s total-elapsed 0h41m19s
epoch 1 batch 524/1,245 processed 16,768 batch-loss 7.5640 grad-norm 120.2388 epoch-elapsed 0h41m22s total-elapsed 0h41m22s
epoch 1 batch 525/1,245 processed 16,800 batch-loss 8.6140 grad-norm 83.5256 epoch-elapsed 0h41m25s total-elapsed 0h41m25s
epoch 1 batch 526/1,245 processed 16,832 batch-loss 5.4774 grad-norm 35.2327 epoch-elapsed 0h41m28s total-elapsed 0h41m28s
epoch 1 batch 527/1,245 processed 16,864 batch-loss 7.4552 grad-norm 171.2855 epoch-elapsed 0h41m31s total-elapsed 0h41m31s
epoch 1 batch 528/1,245 processed 16,896 batch-loss 7.1661 grad-norm 158.7606 epoch-elapsed 0h41m33s total-elapsed 0h41m33s
epoch 1 batch 529/1,245 processed 16,928 batch-loss 7.9698 grad-norm 62.3689 epoch-elapsed 0h41m37s total-elapsed 0h41m37s
epoch 1 batch 530/1,245 processed 16,960 batch-loss 7.2397 grad-norm 65.2580 epoch-elapsed 0h41m41s total-elapsed 0h41m41s
epoch 1 batch 531/1,245 processed 16,992 batch-loss 4.8759 grad-norm 90.6161 epoch-elapsed 0h41m44s total-elapsed 0h41m44s
epoch 1 batch 532/1,245 processed 17,024 batch-loss 7.5493 grad-norm 73.3418 epoch-elapsed 0h41m47s total-elapsed 0h41m47s
epoch 1 batch 533/1,245 processed 17,056 batch-loss 7.3995 grad-norm 125.3553 epoch-elapsed 0h41m50s total-elapsed 0h41m50s
epoch 1 batch 534/1,245 processed 17,088 batch-loss 5.1920 grad-norm 535.6347 epoch-elapsed 0h41m53s total-elapsed 0h41m53s
epoch 1 batch 535/1,245 processed 17,120 batch-loss 5.0986 grad-norm 135.9123 epoch-elapsed 0h41m57s total-elapsed 0h41m57s
epoch 1 batch 536/1,245 processed 17,152 batch-loss 5.8343 grad-norm 59.3363 epoch-elapsed 0h41m59s total-elapsed 0h41m59s
epoch 1 batch 537/1,245 processed 17,184 batch-loss 4.6411 grad-norm 52.1101 epoch-elapsed 0h42m02s total-elapsed 0h42m02s
epoch 1 batch 538/1,245 processed 17,216 batch-loss 4.5031 grad-norm 243.3089 epoch-elapsed 0h42m05s total-elapsed 0h42m05s
epoch 1 batch 539/1,245 processed 17,248 batch-loss 8.5866 grad-norm 314.5148 epoch-elapsed 0h42m09s total-elapsed 0h42m09s
epoch 1 batch 540/1,245 processed 17,280 batch-loss 5.9026 grad-norm 102.6659 epoch-elapsed 0h42m11s total-elapsed 0h42m11s
epoch 1 batch 541/1,245 processed 17,312 batch-loss 6.7547 grad-norm 173.2652 epoch-elapsed 0h42m14s total-elapsed 0h42m14s
epoch 1 batch 542/1,245 processed 17,344 batch-loss 6.4970 grad-norm 316.8577 epoch-elapsed 0h42m17s total-elapsed 0h42m17s
epoch 1 batch 543/1,245 processed 17,376 batch-loss 6.7792 grad-norm 72.1833 epoch-elapsed 0h42m20s total-elapsed 0h42m20s
epoch 1 batch 544/1,245 processed 17,408 batch-loss 9.0685 grad-norm 236.5013 epoch-elapsed 0h42m23s total-elapsed 0h42m23s
epoch 1 batch 545/1,245 processed 17,440 batch-loss 6.9134 grad-norm 111.7664 epoch-elapsed 0h42m25s total-elapsed 0h42m25s
epoch 1 batch 546/1,245 processed 17,472 batch-loss 5.6202 grad-norm 72.2634 epoch-elapsed 0h42m28s total-elapsed 0h42m28s
epoch 1 batch 547/1,245 processed 17,504 batch-loss 7.8608 grad-norm 146.6753 epoch-elapsed 0h42m32s total-elapsed 0h42m32s
epoch 1 batch 548/1,245 processed 17,536 batch-loss 7.4022 grad-norm 172.0155 epoch-elapsed 0h42m36s total-elapsed 0h42m36s
epoch 1 batch 549/1,245 processed 17,568 batch-loss 6.8303 grad-norm 181.4357 epoch-elapsed 0h42m38s total-elapsed 0h42m38s
epoch 1 batch 550/1,245 processed 17,600 batch-loss 4.4430 grad-norm 185.8870 epoch-elapsed 0h42m40s total-elapsed 0h42m40s
epoch 1 batch 551/1,245 processed 17,632 batch-loss 10.7996 grad-norm 176.9970 epoch-elapsed 0h42m45s total-elapsed 0h42m45s
epoch 1 batch 552/1,245 processed 17,664 batch-loss 7.0643 grad-norm 572.4961 epoch-elapsed 0h42m47s total-elapsed 0h42m47s
epoch 1 batch 553/1,245 processed 17,696 batch-loss 4.9612 grad-norm 87.2774 epoch-elapsed 0h42m50s total-elapsed 0h42m50s
epoch 1 batch 554/1,245 processed 17,728 batch-loss 4.9342 grad-norm 44.8712 epoch-elapsed 0h42m52s total-elapsed 0h42m52s
epoch 1 batch 555/1,245 processed 17,760 batch-loss 6.5572 grad-norm 169.1128 epoch-elapsed 0h42m55s total-elapsed 0h42m55s
epoch 1 batch 556/1,245 processed 17,792 batch-loss 7.4442 grad-norm 599.8742 epoch-elapsed 0h42m59s total-elapsed 0h42m59s
epoch 1 batch 557/1,245 processed 17,824 batch-loss 4.3329 grad-norm 86.6673 epoch-elapsed 0h43m01s total-elapsed 0h43m01s
epoch 1 batch 558/1,245 processed 17,856 batch-loss 5.9589 grad-norm 119.7661 epoch-elapsed 0h43m03s total-elapsed 0h43m03s
epoch 1 batch 559/1,245 processed 17,888 batch-loss 7.4995 grad-norm 112.7416 epoch-elapsed 0h43m06s total-elapsed 0h43m06s
epoch 1 batch 560/1,245 processed 17,920 batch-loss 7.8523 grad-norm 100.7157 epoch-elapsed 0h43m10s total-elapsed 0h43m10s
epoch 1 batch 561/1,245 processed 17,952 batch-loss 5.2123 grad-norm 120.8190 epoch-elapsed 0h43m12s total-elapsed 0h43m12s
epoch 1 batch 562/1,245 processed 17,984 batch-loss 5.5832 grad-norm 183.0114 epoch-elapsed 0h43m15s total-elapsed 0h43m15s
epoch 1 batch 563/1,245 processed 18,016 batch-loss 7.1081 grad-norm 117.6272 epoch-elapsed 0h43m17s total-elapsed 0h43m17s
epoch 1 batch 564/1,245 processed 18,048 batch-loss 6.0415 grad-norm 107.1954 epoch-elapsed 0h43m20s total-elapsed 0h43m20s
epoch 1 batch 565/1,245 processed 18,080 batch-loss 6.8529 grad-norm 155.2342 epoch-elapsed 0h43m23s total-elapsed 0h43m23s
epoch 1 batch 566/1,245 processed 18,112 batch-loss 9.2831 grad-norm 884.3144 epoch-elapsed 0h43m28s total-elapsed 0h43m28s
epoch 1 batch 567/1,245 processed 18,144 batch-loss 6.3792 grad-norm 57.3659 epoch-elapsed 0h43m31s total-elapsed 0h43m31s
epoch 1 batch 568/1,245 processed 18,176 batch-loss 9.0655 grad-norm 87.5466 epoch-elapsed 0h43m35s total-elapsed 0h43m35s
epoch 1 batch 569/1,245 processed 18,208 batch-loss 7.6610 grad-norm 2133.4473 epoch-elapsed 0h43m38s total-elapsed 0h43m38s
epoch 1 batch 570/1,245 processed 18,240 batch-loss 6.6102 grad-norm 91.0924 epoch-elapsed 0h43m42s total-elapsed 0h43m42s
epoch 1 batch 571/1,245 processed 18,272 batch-loss 5.5538 grad-norm 388.9337 epoch-elapsed 0h43m46s total-elapsed 0h43m46s
epoch 1 batch 572/1,245 processed 18,304 batch-loss 6.0405 grad-norm 73.7538 epoch-elapsed 0h43m51s total-elapsed 0h43m51s
epoch 1 batch 573/1,245 processed 18,336 batch-loss 6.6937 grad-norm 260.9046 epoch-elapsed 0h43m56s total-elapsed 0h43m56s
epoch 1 batch 574/1,245 processed 18,368 batch-loss 6.6713 grad-norm 212.8413 epoch-elapsed 0h44m01s total-elapsed 0h44m01s
epoch 1 batch 575/1,245 processed 18,400 batch-loss 9.4987 grad-norm 183.2032 epoch-elapsed 0h44m08s total-elapsed 0h44m08s
epoch 1 batch 576/1,245 processed 18,432 batch-loss 6.3051 grad-norm 55.2265 epoch-elapsed 0h44m12s total-elapsed 0h44m12s
epoch 1 batch 577/1,245 processed 18,464 batch-loss 4.1012 grad-norm 85.4816 epoch-elapsed 0h44m16s total-elapsed 0h44m16s
epoch 1 batch 578/1,245 processed 18,496 batch-loss 9.0054 grad-norm 385.4313 epoch-elapsed 0h44m22s total-elapsed 0h44m22s
epoch 1 batch 579/1,245 processed 18,528 batch-loss 5.0262 grad-norm 88.2166 epoch-elapsed 0h44m26s total-elapsed 0h44m26s
epoch 1 batch 580/1,245 processed 18,560 batch-loss 8.3699 grad-norm 177.9760 epoch-elapsed 0h44m32s total-elapsed 0h44m32s
epoch 1 batch 581/1,245 processed 18,592 batch-loss 6.0944 grad-norm 103.9572 epoch-elapsed 0h44m38s total-elapsed 0h44m38s
epoch 1 batch 582/1,245 processed 18,624 batch-loss 6.6101 grad-norm 77.1500 epoch-elapsed 0h44m43s total-elapsed 0h44m43s
epoch 1 batch 583/1,245 processed 18,656 batch-loss 8.7648 grad-norm 107.0701 epoch-elapsed 0h44m49s total-elapsed 0h44m49s
epoch 1 batch 584/1,245 processed 18,688 batch-loss 6.6581 grad-norm 273.5959 epoch-elapsed 0h44m51s total-elapsed 0h44m51s
epoch 1 batch 585/1,245 processed 18,720 batch-loss 5.7032 grad-norm 357.0894 epoch-elapsed 0h44m53s total-elapsed 0h44m53s
epoch 1 batch 586/1,245 processed 18,752 batch-loss 8.2964 grad-norm 106.8757 epoch-elapsed 0h44m57s total-elapsed 0h44m57s
epoch 1 batch 587/1,245 processed 18,784 batch-loss 7.2462 grad-norm 2221.2523 epoch-elapsed 0h44m59s total-elapsed 0h44m59s
epoch 1 batch 588/1,245 processed 18,816 batch-loss 5.2778 grad-norm 414.4426 epoch-elapsed 0h45m02s total-elapsed 0h45m02s
epoch 1 batch 589/1,245 processed 18,848 batch-loss 4.6712 grad-norm 432.2477 epoch-elapsed 0h45m05s total-elapsed 0h45m05s
epoch 1 batch 590/1,245 processed 18,880 batch-loss 5.3075 grad-norm 82.0560 epoch-elapsed 0h45m08s total-elapsed 0h45m08s
epoch 1 batch 591/1,245 processed 18,912 batch-loss 6.6491 grad-norm 118.7974 epoch-elapsed 0h45m11s total-elapsed 0h45m11s
epoch 1 batch 592/1,245 processed 18,944 batch-loss 5.2745 grad-norm 95.0573 epoch-elapsed 0h45m14s total-elapsed 0h45m14s
epoch 1 batch 593/1,245 processed 18,976 batch-loss 6.3192 grad-norm 44.2687 epoch-elapsed 0h45m16s total-elapsed 0h45m16s
epoch 1 batch 594/1,245 processed 19,008 batch-loss 5.4183 grad-norm 120.6363 epoch-elapsed 0h45m19s total-elapsed 0h45m19s
epoch 1 batch 595/1,245 processed 19,040 batch-loss 7.5380 grad-norm 104.6314 epoch-elapsed 0h45m22s total-elapsed 0h45m22s
epoch 1 batch 596/1,245 processed 19,072 batch-loss 7.4121 grad-norm 251.3215 epoch-elapsed 0h45m26s total-elapsed 0h45m26s
epoch 1 batch 597/1,245 processed 19,104 batch-loss 5.6162 grad-norm 60.2500 epoch-elapsed 0h45m28s total-elapsed 0h45m28s
epoch 1 batch 598/1,245 processed 19,136 batch-loss 7.9898 grad-norm 451.9214 epoch-elapsed 0h45m32s total-elapsed 0h45m32s
epoch 1 batch 599/1,245 processed 19,168 batch-loss 4.7017 grad-norm 106.2936 epoch-elapsed 0h45m34s total-elapsed 0h45m34s
epoch 1 batch 600/1,245 processed 19,200 batch-loss 4.9391 grad-norm 1310.0097 epoch-elapsed 0h45m36s total-elapsed 0h45m36s
epoch 1 batch 601/1,245 processed 19,232 batch-loss 4.4885 grad-norm 36.1093 epoch-elapsed 0h45m39s total-elapsed 0h45m39s
epoch 1 batch 602/1,245 processed 19,264 batch-loss 5.3887 grad-norm 138.0292 epoch-elapsed 0h45m42s total-elapsed 0h45m42s
epoch 1 batch 603/1,245 processed 19,296 batch-loss 6.3117 grad-norm 480.9025 epoch-elapsed 0h45m45s total-elapsed 0h45m45s
epoch 1 batch 604/1,245 processed 19,328 batch-loss 6.0473 grad-norm 162.6600 epoch-elapsed 0h45m47s total-elapsed 0h45m47s
epoch 1 batch 605/1,245 processed 19,360 batch-loss 7.1252 grad-norm 144.9712 epoch-elapsed 0h45m51s total-elapsed 0h45m51s
epoch 1 batch 606/1,245 processed 19,392 batch-loss 8.3536 grad-norm 433.6059 epoch-elapsed 0h45m55s total-elapsed 0h45m55s
epoch 1 batch 607/1,245 processed 19,424 batch-loss 4.9411 grad-norm 41.2738 epoch-elapsed 0h46m01s total-elapsed 0h46m01s
epoch 1 batch 608/1,245 processed 19,456 batch-loss 8.0399 grad-norm 175.2017 epoch-elapsed 0h46m06s total-elapsed 0h46m06s
epoch 1 batch 609/1,245 processed 19,488 batch-loss 4.0820 grad-norm 84.6372 epoch-elapsed 0h46m09s total-elapsed 0h46m09s
epoch 1 batch 610/1,245 processed 19,520 batch-loss 6.4112 grad-norm 81.2229 epoch-elapsed 0h46m14s total-elapsed 0h46m14s
epoch 1 batch 611/1,245 processed 19,552 batch-loss 7.2158 grad-norm 122.3232 epoch-elapsed 0h46m18s total-elapsed 0h46m18s
epoch 1 batch 612/1,245 processed 19,584 batch-loss 5.9229 grad-norm 143.1458 epoch-elapsed 0h46m22s total-elapsed 0h46m22s
epoch 1 batch 613/1,245 processed 19,616 batch-loss 7.9155 grad-norm 104.6874 epoch-elapsed 0h46m28s total-elapsed 0h46m28s
epoch 1 batch 614/1,245 processed 19,648 batch-loss 9.2337 grad-norm 242.2835 epoch-elapsed 0h46m34s total-elapsed 0h46m34s
epoch 1 batch 615/1,245 processed 19,680 batch-loss 8.1380 grad-norm 834.5944 epoch-elapsed 0h46m40s total-elapsed 0h46m40s
epoch 1 batch 616/1,245 processed 19,712 batch-loss 6.6970 grad-norm 133.5016 epoch-elapsed 0h46m45s total-elapsed 0h46m45s
epoch 1 batch 617/1,245 processed 19,744 batch-loss 5.8799 grad-norm 57.2024 epoch-elapsed 0h46m50s total-elapsed 0h46m50s
epoch 1 batch 618/1,245 processed 19,776 batch-loss 4.9360 grad-norm 403.7813 epoch-elapsed 0h46m53s total-elapsed 0h46m53s
epoch 1 batch 619/1,245 processed 19,808 batch-loss 5.7890 grad-norm 152.0620 epoch-elapsed 0h46m56s total-elapsed 0h46m56s
epoch 1 batch 620/1,245 processed 19,840 batch-loss 6.4677 grad-norm 228.7463 epoch-elapsed 0h47m00s total-elapsed 0h47m00s
epoch 1 batch 621/1,245 processed 19,872 batch-loss 4.6231 grad-norm 83.3019 epoch-elapsed 0h47m03s total-elapsed 0h47m03s
epoch 1 batch 622/1,245 processed 19,904 batch-loss 7.3300 grad-norm 769.6707 epoch-elapsed 0h47m09s total-elapsed 0h47m09s
epoch 1 batch 623/1,245 processed 19,936 batch-loss 8.1820 grad-norm 314.0855 epoch-elapsed 0h47m14s total-elapsed 0h47m14s
dev-fscore (Recall=85.52, Precision=92.73, FScore=88.98, CompleteMatch=21.94) dev-elapsed 0h02m59s total-elapsed 0h50m13s
Removing previous model file models/en_multibert_empty_l2_dev=94.94.pt_dev=85.73.pt...
Saving new best model to models/en_multibert_empty_l2_dev=94.94.pt_dev=88.98...
reindex-elapsed: 0h09m57s
Saving recomputed index
save-elapsed: 0h02m38s
epoch 1 batch 624/1,245 processed 19,968 batch-loss 19.2678 grad-norm 327.1591 epoch-elapsed 1h03m03s total-elapsed 1h03m03s
epoch 1 batch 625/1,245 processed 20,000 batch-loss 17.7235 grad-norm 904.5539 epoch-elapsed 1h03m06s total-elapsed 1h03m06s
epoch 1 batch 626/1,245 processed 20,032 batch-loss 20.8446 grad-norm 628.4408 epoch-elapsed 1h03m09s total-elapsed 1h03m09s
epoch 1 batch 627/1,245 processed 20,064 batch-loss 13.6492 grad-norm 47.6655 epoch-elapsed 1h03m11s total-elapsed 1h03m11s
epoch 1 batch 628/1,245 processed 20,096 batch-loss 15.7302 grad-norm 154.0688 epoch-elapsed 1h03m14s total-elapsed 1h03m14s
epoch 1 batch 629/1,245 processed 20,128 batch-loss 15.9020 grad-norm 35.2721 epoch-elapsed 1h03m16s total-elapsed 1h03m16s
epoch 1 batch 630/1,245 processed 20,160 batch-loss 20.4930 grad-norm 127.6101 epoch-elapsed 1h03m20s total-elapsed 1h03m20s
epoch 1 batch 631/1,245 processed 20,192 batch-loss 22.3559 grad-norm 312.8933 epoch-elapsed 1h03m23s total-elapsed 1h03m23s
epoch 1 batch 632/1,245 processed 20,224 batch-loss 16.6022 grad-norm 38.8928 epoch-elapsed 1h03m25s total-elapsed 1h03m25s
epoch 1 batch 633/1,245 processed 20,256 batch-loss 22.5348 grad-norm 131.9182 epoch-elapsed 1h03m28s total-elapsed 1h03m28s
epoch 1 batch 634/1,245 processed 20,288 batch-loss 28.9754 grad-norm 259.6811 epoch-elapsed 1h03m30s total-elapsed 1h03m30s
epoch 1 batch 635/1,245 processed 20,320 batch-loss 19.1514 grad-norm 77.4395 epoch-elapsed 1h03m32s total-elapsed 1h03m32s
epoch 1 batch 636/1,245 processed 20,352 batch-loss 26.1657 grad-norm 29.2523 epoch-elapsed 1h03m35s total-elapsed 1h03m35s
epoch 1 batch 637/1,245 processed 20,384 batch-loss 20.1679 grad-norm 544.2459 epoch-elapsed 1h03m37s total-elapsed 1h03m37s
epoch 1 batch 638/1,245 processed 20,416 batch-loss 19.8828 grad-norm 26.8655 epoch-elapsed 1h03m39s total-elapsed 1h03m39s
epoch 1 batch 639/1,245 processed 20,448 batch-loss 27.0855 grad-norm 1020.2213 epoch-elapsed 1h03m42s total-elapsed 1h03m42s
epoch 1 batch 640/1,245 processed 20,480 batch-loss 24.4585 grad-norm 469.5157 epoch-elapsed 1h03m44s total-elapsed 1h03m44s
epoch 1 batch 641/1,245 processed 20,512 batch-loss 22.8492 grad-norm 97.8474 epoch-elapsed 1h03m47s total-elapsed 1h03m47s
epoch 1 batch 642/1,245 processed 20,544 batch-loss 28.3599 grad-norm 397.5501 epoch-elapsed 1h03m50s total-elapsed 1h03m50s
epoch 1 batch 643/1,245 processed 20,576 batch-loss 23.4493 grad-norm 46.6899 epoch-elapsed 1h03m53s total-elapsed 1h03m53s
epoch 1 batch 644/1,245 processed 20,608 batch-loss 21.8975 grad-norm 211.0349 epoch-elapsed 1h03m56s total-elapsed 1h03m56s
epoch 1 batch 645/1,245 processed 20,640 batch-loss 19.9361 grad-norm 90.0815 epoch-elapsed 1h03m58s total-elapsed 1h03m58s
epoch 1 batch 646/1,245 processed 20,672 batch-loss 21.8314 grad-norm 247.6047 epoch-elapsed 1h04m01s total-elapsed 1h04m01s
epoch 1 batch 647/1,245 processed 20,704 batch-loss 25.6322 grad-norm 247.1232 epoch-elapsed 1h04m05s total-elapsed 1h04m05s
epoch 1 batch 648/1,245 processed 20,736 batch-loss 25.7131 grad-norm 839.8205 epoch-elapsed 1h04m08s total-elapsed 1h04m08s
epoch 1 batch 649/1,245 processed 20,768 batch-loss 20.0997 grad-norm 104.4762 epoch-elapsed 1h04m11s total-elapsed 1h04m11s
epoch 1 batch 650/1,245 processed 20,800 batch-loss 25.9056 grad-norm 401.9583 epoch-elapsed 1h04m15s total-elapsed 1h04m15s
epoch 1 batch 651/1,245 processed 20,832 batch-loss 24.7351 grad-norm 325.2198 epoch-elapsed 1h04m18s total-elapsed 1h04m18s
epoch 1 batch 652/1,245 processed 20,864 batch-loss 30.0415 grad-norm 255.0790 epoch-elapsed 1h04m23s total-elapsed 1h04m23s
epoch 1 batch 653/1,245 processed 20,896 batch-loss 31.4153 grad-norm 444.9786 epoch-elapsed 1h04m26s total-elapsed 1h04m26s
epoch 1 batch 654/1,245 processed 20,928 batch-loss 25.0800 grad-norm 131.2235 epoch-elapsed 1h04m29s total-elapsed 1h04m29s
epoch 1 batch 655/1,245 processed 20,960 batch-loss 25.5635 grad-norm 69.0201 epoch-elapsed 1h04m32s total-elapsed 1h04m32s
epoch 1 batch 656/1,245 processed 20,992 batch-loss 23.6364 grad-norm 239.5406 epoch-elapsed 1h04m35s total-elapsed 1h04m35s
epoch 1 batch 657/1,245 processed 21,024 batch-loss 24.0299 grad-norm 656.3103 epoch-elapsed 1h04m38s total-elapsed 1h04m38s
epoch 1 batch 658/1,245 processed 21,056 batch-loss 19.8906 grad-norm 32.7303 epoch-elapsed 1h04m40s total-elapsed 1h04m40s
epoch 1 batch 659/1,245 processed 21,088 batch-loss 18.2077 grad-norm 29.3561 epoch-elapsed 1h04m43s total-elapsed 1h04m43s
epoch 1 batch 660/1,245 processed 21,120 batch-loss 27.2972 grad-norm 137.3697 epoch-elapsed 1h04m46s total-elapsed 1h04m46s
epoch 1 batch 661/1,245 processed 21,152 batch-loss 21.7996 grad-norm 244.3699 epoch-elapsed 1h04m48s total-elapsed 1h04m48s
epoch 1 batch 662/1,245 processed 21,184 batch-loss 25.6557 grad-norm 39.5919 epoch-elapsed 1h04m53s total-elapsed 1h04m53s
epoch 1 batch 663/1,245 processed 21,216 batch-loss 21.0175 grad-norm 624.6769 epoch-elapsed 1h04m56s total-elapsed 1h04m56s
epoch 1 batch 664/1,245 processed 21,248 batch-loss 17.6417 grad-norm 33.2575 epoch-elapsed 1h04m58s total-elapsed 1h04m58s
epoch 1 batch 665/1,245 processed 21,280 batch-loss 24.1432 grad-norm 40.7693 epoch-elapsed 1h05m01s total-elapsed 1h05m01s
epoch 1 batch 666/1,245 processed 21,312 batch-loss 19.4146 grad-norm 163.8256 epoch-elapsed 1h05m04s total-elapsed 1h05m04s
epoch 1 batch 667/1,245 processed 21,344 batch-loss 17.5309 grad-norm 106.7182 epoch-elapsed 1h05m07s total-elapsed 1h05m07s
epoch 1 batch 668/1,245 processed 21,376 batch-loss 18.0320 grad-norm 53.3580 epoch-elapsed 1h05m10s total-elapsed 1h05m10s
epoch 1 batch 669/1,245 processed 21,408 batch-loss 19.4692 grad-norm 1219.0894 epoch-elapsed 1h05m12s total-elapsed 1h05m12s
epoch 1 batch 670/1,245 processed 21,440 batch-loss 22.6941 grad-norm 45.1966 epoch-elapsed 1h05m16s total-elapsed 1h05m16s
epoch 1 batch 671/1,245 processed 21,472 batch-loss 20.1142 grad-norm 42.3249 epoch-elapsed 1h05m18s total-elapsed 1h05m18s
epoch 1 batch 672/1,245 processed 21,504 batch-loss 19.9894 grad-norm 35.1214 epoch-elapsed 1h05m21s total-elapsed 1h05m21s
epoch 1 batch 673/1,245 processed 21,536 batch-loss 25.8047 grad-norm 47.4187 epoch-elapsed 1h05m24s total-elapsed 1h05m24s
epoch 1 batch 674/1,245 processed 21,568 batch-loss 27.6973 grad-norm 71.8720 epoch-elapsed 1h05m28s total-elapsed 1h05m28s
epoch 1 batch 675/1,245 processed 21,600 batch-loss 18.4638 grad-norm 235.3392 epoch-elapsed 1h05m30s total-elapsed 1h05m30s
epoch 1 batch 676/1,245 processed 21,632 batch-loss 20.9189 grad-norm 67.3343 epoch-elapsed 1h05m33s total-elapsed 1h05m33s
epoch 1 batch 677/1,245 processed 21,664 batch-loss 14.7123 grad-norm 235.8818 epoch-elapsed 1h05m35s total-elapsed 1h05m35s
epoch 1 batch 678/1,245 processed 21,696 batch-loss 23.4870 grad-norm 153.4431 epoch-elapsed 1h05m38s total-elapsed 1h05m38s
epoch 1 batch 679/1,245 processed 21,728 batch-loss 19.4383 grad-norm 156.1498 epoch-elapsed 1h05m41s total-elapsed 1h05m41s
epoch 1 batch 680/1,245 processed 21,760 batch-loss 15.5416 grad-norm 657.2333 epoch-elapsed 1h05m43s total-elapsed 1h05m43s
epoch 1 batch 681/1,245 processed 21,792 batch-loss 12.4432 grad-norm 114.4444 epoch-elapsed 1h05m45s total-elapsed 1h05m45s
epoch 1 batch 682/1,245 processed 21,824 batch-loss 16.5783 grad-norm 189.3533 epoch-elapsed 1h05m48s total-elapsed 1h05m48s
epoch 1 batch 683/1,245 processed 21,856 batch-loss 18.3160 grad-norm 383.9771 epoch-elapsed 1h05m52s total-elapsed 1h05m52s
epoch 1 batch 684/1,245 processed 21,888 batch-loss 18.1590 grad-norm 256.7282 epoch-elapsed 1h05m55s total-elapsed 1h05m55s
epoch 1 batch 685/1,245 processed 21,920 batch-loss 15.8391 grad-norm 198.3513 epoch-elapsed 1h05m57s total-elapsed 1h05m57s
epoch 1 batch 686/1,245 processed 21,952 batch-loss 17.1865 grad-norm 132.2787 epoch-elapsed 1h06m00s total-elapsed 1h06m00s
epoch 1 batch 687/1,245 processed 21,984 batch-loss 16.9192 grad-norm 175.5139 epoch-elapsed 1h06m02s total-elapsed 1h06m02s
epoch 1 batch 688/1,245 processed 22,016 batch-loss 16.6232 grad-norm 58.4487 epoch-elapsed 1h06m05s total-elapsed 1h06m05s
epoch 1 batch 689/1,245 processed 22,048 batch-loss 12.1324 grad-norm 85.9253 epoch-elapsed 1h06m06s total-elapsed 1h06m06s
epoch 1 batch 690/1,245 processed 22,080 batch-loss 12.8329 grad-norm 64.6956 epoch-elapsed 1h06m09s total-elapsed 1h06m09s
epoch 1 batch 691/1,245 processed 22,112 batch-loss 13.1201 grad-norm 36.5638 epoch-elapsed 1h06m11s total-elapsed 1h06m11s
epoch 1 batch 692/1,245 processed 22,144 batch-loss 19.9796 grad-norm 846.4727 epoch-elapsed 1h06m15s total-elapsed 1h06m15s
epoch 1 batch 693/1,245 processed 22,176 batch-loss 19.5233 grad-norm 173.6130 epoch-elapsed 1h06m18s total-elapsed 1h06m18s
epoch 1 batch 694/1,245 processed 22,208 batch-loss 16.0375 grad-norm 67.1322 epoch-elapsed 1h06m20s total-elapsed 1h06m20s
epoch 1 batch 695/1,245 processed 22,240 batch-loss 18.2412 grad-norm 46.8678 epoch-elapsed 1h06m23s total-elapsed 1h06m23s
epoch 1 batch 696/1,245 processed 22,272 batch-loss 14.1439 grad-norm 593.8752 epoch-elapsed 1h06m25s total-elapsed 1h06m25s
epoch 1 batch 697/1,245 processed 22,304 batch-loss 16.9532 grad-norm 106.4499 epoch-elapsed 1h06m27s total-elapsed 1h06m27s
epoch 1 batch 698/1,245 processed 22,336 batch-loss 11.9537 grad-norm 358.4467 epoch-elapsed 1h06m30s total-elapsed 1h06m30s
epoch 1 batch 699/1,245 processed 22,368 batch-loss 14.4517 grad-norm 916.5786 epoch-elapsed 1h06m33s total-elapsed 1h06m33s
epoch 1 batch 700/1,245 processed 22,400 batch-loss 16.3613 grad-norm 81.4569 epoch-elapsed 1h06m35s total-elapsed 1h06m35s
epoch 1 batch 701/1,245 processed 22,432 batch-loss 17.4131 grad-norm 233.4493 epoch-elapsed 1h06m38s total-elapsed 1h06m38s
epoch 1 batch 702/1,245 processed 22,464 batch-loss 16.9455 grad-norm 44.6139 epoch-elapsed 1h06m41s total-elapsed 1h06m41s
epoch 1 batch 703/1,245 processed 22,496 batch-loss 19.8103 grad-norm 106.3261 epoch-elapsed 1h06m45s total-elapsed 1h06m45s
epoch 1 batch 704/1,245 processed 22,528 batch-loss 14.8001 grad-norm 54.4867 epoch-elapsed 1h06m48s total-elapsed 1h06m48s
epoch 1 batch 705/1,245 processed 22,560 batch-loss 18.6491 grad-norm 90.8515 epoch-elapsed 1h06m52s total-elapsed 1h06m52s
epoch 1 batch 706/1,245 processed 22,592 batch-loss 19.0132 grad-norm 120.9975 epoch-elapsed 1h06m55s total-elapsed 1h06m55s
epoch 1 batch 707/1,245 processed 22,624 batch-loss 11.7856 grad-norm 462.1012 epoch-elapsed 1h06m57s total-elapsed 1h06m57s
epoch 1 batch 708/1,245 processed 22,656 batch-loss 14.5357 grad-norm 71.1487 epoch-elapsed 1h07m01s total-elapsed 1h07m01s
epoch 1 batch 709/1,245 processed 22,688 batch-loss 10.5902 grad-norm 59.6955 epoch-elapsed 1h07m03s total-elapsed 1h07m03s
epoch 1 batch 710/1,245 processed 22,720 batch-loss 17.4496 grad-norm 75.0783 epoch-elapsed 1h07m06s total-elapsed 1h07m06s
epoch 1 batch 711/1,245 processed 22,752 batch-loss 19.1997 grad-norm 199.6044 epoch-elapsed 1h07m11s total-elapsed 1h07m11s
epoch 1 batch 712/1,245 processed 22,784 batch-loss 16.4620 grad-norm 85.3306 epoch-elapsed 1h07m13s total-elapsed 1h07m13s
epoch 1 batch 713/1,245 processed 22,816 batch-loss 15.1169 grad-norm 218.1104 epoch-elapsed 1h07m16s total-elapsed 1h07m16s
epoch 1 batch 714/1,245 processed 22,848 batch-loss 12.5755 grad-norm 132.4781 epoch-elapsed 1h07m19s total-elapsed 1h07m19s
epoch 1 batch 715/1,245 processed 22,880 batch-loss 12.2249 grad-norm 88.2686 epoch-elapsed 1h07m21s total-elapsed 1h07m21s
epoch 1 batch 716/1,245 processed 22,912 batch-loss 16.4502 grad-norm 45.3336 epoch-elapsed 1h07m25s total-elapsed 1h07m25s
epoch 1 batch 717/1,245 processed 22,944 batch-loss 15.5291 grad-norm 184.7259 epoch-elapsed 1h07m28s total-elapsed 1h07m28s
epoch 1 batch 718/1,245 processed 22,976 batch-loss 11.5861 grad-norm 45.4104 epoch-elapsed 1h07m32s total-elapsed 1h07m32s
epoch 1 batch 719/1,245 processed 23,008 batch-loss 13.1310 grad-norm 245.5049 epoch-elapsed 1h07m36s total-elapsed 1h07m36s
epoch 1 batch 720/1,245 processed 23,040 batch-loss 12.4063 grad-norm 86.6492 epoch-elapsed 1h07m40s total-elapsed 1h07m40s
epoch 1 batch 721/1,245 processed 23,072 batch-loss 11.1379 grad-norm 53.9079 epoch-elapsed 1h07m43s total-elapsed 1h07m43s
epoch 1 batch 722/1,245 processed 23,104 batch-loss 13.9497 grad-norm 57.0098 epoch-elapsed 1h07m46s total-elapsed 1h07m46s
epoch 1 batch 723/1,245 processed 23,136 batch-loss 17.5005 grad-norm 209.3772 epoch-elapsed 1h07m49s total-elapsed 1h07m49s
epoch 1 batch 724/1,245 processed 23,168 batch-loss 12.7795 grad-norm 195.7878 epoch-elapsed 1h07m51s total-elapsed 1h07m51s
epoch 1 batch 725/1,245 processed 23,200 batch-loss 14.4838 grad-norm 128.1700 epoch-elapsed 1h07m55s total-elapsed 1h07m55s
epoch 1 batch 726/1,245 processed 23,232 batch-loss 15.1583 grad-norm 661.3130 epoch-elapsed 1h07m58s total-elapsed 1h07m58s
epoch 1 batch 727/1,245 processed 23,264 batch-loss 14.5726 grad-norm 51.8829 epoch-elapsed 1h08m02s total-elapsed 1h08m02s
epoch 1 batch 728/1,245 processed 23,296 batch-loss 11.7896 grad-norm 112.3429 epoch-elapsed 1h08m05s total-elapsed 1h08m05s
epoch 1 batch 729/1,245 processed 23,328 batch-loss 10.7639 grad-norm 71.2291 epoch-elapsed 1h08m07s total-elapsed 1h08m07s
epoch 1 batch 730/1,245 processed 23,360 batch-loss 13.4769 grad-norm 119.0887 epoch-elapsed 1h08m11s total-elapsed 1h08m11s
epoch 1 batch 731/1,245 processed 23,392 batch-loss 9.3986 grad-norm 34.1653 epoch-elapsed 1h08m12s total-elapsed 1h08m12s
epoch 1 batch 732/1,245 processed 23,424 batch-loss 7.5222 grad-norm 74.1035 epoch-elapsed 1h08m15s total-elapsed 1h08m15s
epoch 1 batch 733/1,245 processed 23,456 batch-loss 9.7510 grad-norm 109.3397 epoch-elapsed 1h08m17s total-elapsed 1h08m17s
epoch 1 batch 734/1,245 processed 23,488 batch-loss 11.9582 grad-norm 119.9906 epoch-elapsed 1h08m20s total-elapsed 1h08m20s
epoch 1 batch 735/1,245 processed 23,520 batch-loss 8.8306 grad-norm 88.1101 epoch-elapsed 1h08m23s total-elapsed 1h08m23s
epoch 1 batch 736/1,245 processed 23,552 batch-loss 11.6992 grad-norm 108.4223 epoch-elapsed 1h08m25s total-elapsed 1h08m25s
epoch 1 batch 737/1,245 processed 23,584 batch-loss 11.0654 grad-norm 221.9582 epoch-elapsed 1h08m28s total-elapsed 1h08m28s
epoch 1 batch 738/1,245 processed 23,616 batch-loss 5.8341 grad-norm 39.0806 epoch-elapsed 1h08m30s total-elapsed 1h08m30s
epoch 1 batch 739/1,245 processed 23,648 batch-loss 12.8365 grad-norm 133.8577 epoch-elapsed 1h08m34s total-elapsed 1h08m34s
epoch 1 batch 740/1,245 processed 23,680 batch-loss 7.7951 grad-norm 71.6661 epoch-elapsed 1h08m36s total-elapsed 1h08m36s
epoch 1 batch 741/1,245 processed 23,712 batch-loss 16.4564 grad-norm 263.8395 epoch-elapsed 1h08m40s total-elapsed 1h08m40s
epoch 1 batch 742/1,245 processed 23,744 batch-loss 12.4135 grad-norm 1370.1631 epoch-elapsed 1h08m43s total-elapsed 1h08m43s
epoch 1 batch 743/1,245 processed 23,776 batch-loss 10.5637 grad-norm 132.5722 epoch-elapsed 1h08m47s total-elapsed 1h08m47s
epoch 1 batch 744/1,245 processed 23,808 batch-loss 13.1911 grad-norm 63.2338 epoch-elapsed 1h08m50s total-elapsed 1h08m50s
epoch 1 batch 745/1,245 processed 23,840 batch-loss 10.7227 grad-norm 368.8945 epoch-elapsed 1h08m53s total-elapsed 1h08m53s
epoch 1 batch 746/1,245 processed 23,872 batch-loss 10.9874 grad-norm 130.4968 epoch-elapsed 1h08m55s total-elapsed 1h08m55s
epoch 1 batch 747/1,245 processed 23,904 batch-loss 10.0791 grad-norm 248.3797 epoch-elapsed 1h08m58s total-elapsed 1h08m58s
epoch 1 batch 748/1,245 processed 23,936 batch-loss 10.1419 grad-norm 140.3843 epoch-elapsed 1h09m01s total-elapsed 1h09m01s
epoch 1 batch 749/1,245 processed 23,968 batch-loss 12.0021 grad-norm 212.4842 epoch-elapsed 1h09m04s total-elapsed 1h09m04s
epoch 1 batch 750/1,245 processed 24,000 batch-loss 10.1101 grad-norm 138.2073 epoch-elapsed 1h09m07s total-elapsed 1h09m07s
epoch 1 batch 751/1,245 processed 24,032 batch-loss 6.9391 grad-norm 74.7779 epoch-elapsed 1h09m09s total-elapsed 1h09m09s
epoch 1 batch 752/1,245 processed 24,064 batch-loss 10.5398 grad-norm 67.8573 epoch-elapsed 1h09m13s total-elapsed 1h09m13s
epoch 1 batch 753/1,245 processed 24,096 batch-loss 9.0111 grad-norm 91.5620 epoch-elapsed 1h09m16s total-elapsed 1h09m16s
epoch 1 batch 754/1,245 processed 24,128 batch-loss 12.9998 grad-norm 114.9583 epoch-elapsed 1h09m20s total-elapsed 1h09m20s
epoch 1 batch 755/1,245 processed 24,160 batch-loss 8.9648 grad-norm 179.3345 epoch-elapsed 1h09m22s total-elapsed 1h09m22s
epoch 1 batch 756/1,245 processed 24,192 batch-loss 8.6893 grad-norm 99.3620 epoch-elapsed 1h09m25s total-elapsed 1h09m25s
epoch 1 batch 757/1,245 processed 24,224 batch-loss 10.9412 grad-norm 245.9861 epoch-elapsed 1h09m28s total-elapsed 1h09m28s
epoch 1 batch 758/1,245 processed 24,256 batch-loss 9.7384 grad-norm 715.0221 epoch-elapsed 1h09m30s total-elapsed 1h09m30s
epoch 1 batch 759/1,245 processed 24,288 batch-loss 8.0059 grad-norm 146.0520 epoch-elapsed 1h09m32s total-elapsed 1h09m32s
epoch 1 batch 760/1,245 processed 24,320 batch-loss 8.8252 grad-norm 142.9690 epoch-elapsed 1h09m35s total-elapsed 1h09m35s
epoch 1 batch 761/1,245 processed 24,352 batch-loss 9.1656 grad-norm 231.7183 epoch-elapsed 1h09m38s total-elapsed 1h09m38s
epoch 1 batch 762/1,245 processed 24,384 batch-loss 9.6076 grad-norm 139.9632 epoch-elapsed 1h09m40s total-elapsed 1h09m40s
epoch 1 batch 763/1,245 processed 24,416 batch-loss 9.5033 grad-norm 74.7364 epoch-elapsed 1h09m44s total-elapsed 1h09m44s
epoch 1 batch 764/1,245 processed 24,448 batch-loss 9.3863 grad-norm 217.5506 epoch-elapsed 1h09m47s total-elapsed 1h09m47s
epoch 1 batch 765/1,245 processed 24,480 batch-loss 9.4327 grad-norm 484.6707 epoch-elapsed 1h09m49s total-elapsed 1h09m49s
epoch 1 batch 766/1,245 processed 24,512 batch-loss 10.1994 grad-norm 92.1144 epoch-elapsed 1h09m52s total-elapsed 1h09m52s
epoch 1 batch 767/1,245 processed 24,544 batch-loss 8.8162 grad-norm 403.3968 epoch-elapsed 1h09m55s total-elapsed 1h09m55s
epoch 1 batch 768/1,245 processed 24,576 batch-loss 7.7028 grad-norm 95.3427 epoch-elapsed 1h09m57s total-elapsed 1h09m57s
epoch 1 batch 769/1,245 processed 24,608 batch-loss 12.9932 grad-norm 220.9852 epoch-elapsed 1h10m00s total-elapsed 1h10m00s
epoch 1 batch 770/1,245 processed 24,640 batch-loss 8.9400 grad-norm 163.6087 epoch-elapsed 1h10m03s total-elapsed 1h10m03s
epoch 1 batch 771/1,245 processed 24,672 batch-loss 8.6757 grad-norm 375.4128 epoch-elapsed 1h10m06s total-elapsed 1h10m06s
epoch 1 batch 772/1,245 processed 24,704 batch-loss 7.6991 grad-norm 57.5991 epoch-elapsed 1h10m09s total-elapsed 1h10m09s
epoch 1 batch 773/1,245 processed 24,736 batch-loss 11.2378 grad-norm 92.0656 epoch-elapsed 1h10m12s total-elapsed 1h10m12s
epoch 1 batch 774/1,245 processed 24,768 batch-loss 11.9831 grad-norm 329.1941 epoch-elapsed 1h10m16s total-elapsed 1h10m16s
epoch 1 batch 775/1,245 processed 24,800 batch-loss 9.5523 grad-norm 388.1309 epoch-elapsed 1h10m19s total-elapsed 1h10m19s
epoch 1 batch 776/1,245 processed 24,832 batch-loss 8.9316 grad-norm 137.6853 epoch-elapsed 1h10m21s total-elapsed 1h10m21s
epoch 1 batch 777/1,245 processed 24,864 batch-loss 6.7583 grad-norm 51.2140 epoch-elapsed 1h10m24s total-elapsed 1h10m24s
epoch 1 batch 778/1,245 processed 24,896 batch-loss 9.0889 grad-norm 424.8572 epoch-elapsed 1h10m27s total-elapsed 1h10m27s
epoch 1 batch 779/1,245 processed 24,928 batch-loss 9.8498 grad-norm 56.7317 epoch-elapsed 1h10m30s total-elapsed 1h10m30s
epoch 1 batch 780/1,245 processed 24,960 batch-loss 8.5860 grad-norm 128.5034 epoch-elapsed 1h10m32s total-elapsed 1h10m32s
epoch 1 batch 781/1,245 processed 24,992 batch-loss 9.3231 grad-norm 319.4353 epoch-elapsed 1h10m35s total-elapsed 1h10m35s
epoch 1 batch 782/1,245 processed 25,024 batch-loss 8.9282 grad-norm 310.2683 epoch-elapsed 1h10m38s total-elapsed 1h10m38s
epoch 1 batch 783/1,245 processed 25,056 batch-loss 9.0291 grad-norm 120.7373 epoch-elapsed 1h10m41s total-elapsed 1h10m41s
epoch 1 batch 784/1,245 processed 25,088 batch-loss 9.2360 grad-norm 703.8227 epoch-elapsed 1h10m43s total-elapsed 1h10m43s
epoch 1 batch 785/1,245 processed 25,120 batch-loss 11.5511 grad-norm 97.1031 epoch-elapsed 1h10m46s total-elapsed 1h10m46s
epoch 1 batch 786/1,245 processed 25,152 batch-loss 7.5634 grad-norm 353.9258 epoch-elapsed 1h10m49s total-elapsed 1h10m49s
epoch 1 batch 787/1,245 processed 25,184 batch-loss 10.3008 grad-norm 119.8938 epoch-elapsed 1h10m52s total-elapsed 1h10m52s
epoch 1 batch 788/1,245 processed 25,216 batch-loss 8.7390 grad-norm 97.8032 epoch-elapsed 1h10m54s total-elapsed 1h10m54s
epoch 1 batch 789/1,245 processed 25,248 batch-loss 9.1541 grad-norm 344.4475 epoch-elapsed 1h10m57s total-elapsed 1h10m57s
epoch 1 batch 790/1,245 processed 25,280 batch-loss 12.4944 grad-norm 144.3300 epoch-elapsed 1h11m01s total-elapsed 1h11m01s
epoch 1 batch 791/1,245 processed 25,312 batch-loss 10.5082 grad-norm 539.3435 epoch-elapsed 1h11m03s total-elapsed 1h11m03s
epoch 1 batch 792/1,245 processed 25,344 batch-loss 8.9645 grad-norm 70.4397 epoch-elapsed 1h11m06s total-elapsed 1h11m06s
epoch 1 batch 793/1,245 processed 25,376 batch-loss 9.6113 grad-norm 197.8676 epoch-elapsed 1h11m10s total-elapsed 1h11m10s
epoch 1 batch 794/1,245 processed 25,408 batch-loss 9.6012 grad-norm 796.9236 epoch-elapsed 1h11m12s total-elapsed 1h11m12s
epoch 1 batch 795/1,245 processed 25,440 batch-loss 9.5757 grad-norm 78.9627 epoch-elapsed 1h11m14s total-elapsed 1h11m14s
epoch 1 batch 796/1,245 processed 25,472 batch-loss 6.9313 grad-norm 82.5609 epoch-elapsed 1h11m16s total-elapsed 1h11m16s
epoch 1 batch 797/1,245 processed 25,504 batch-loss 12.0693 grad-norm 229.0419 epoch-elapsed 1h11m20s total-elapsed 1h11m20s
epoch 1 batch 798/1,245 processed 25,536 batch-loss 12.5442 grad-norm 1383.8577 epoch-elapsed 1h11m23s total-elapsed 1h11m23s
epoch 1 batch 799/1,245 processed 25,568 batch-loss 10.0293 grad-norm 173.8603 epoch-elapsed 1h11m26s total-elapsed 1h11m26s
epoch 1 batch 800/1,245 processed 25,600 batch-loss 9.9735 grad-norm 120.7486 epoch-elapsed 1h11m29s total-elapsed 1h11m29s
epoch 1 batch 801/1,245 processed 25,632 batch-loss 7.5865 grad-norm 81.7251 epoch-elapsed 1h11m31s total-elapsed 1h11m31s
epoch 1 batch 802/1,245 processed 25,664 batch-loss 9.3775 grad-norm 285.4753 epoch-elapsed 1h11m33s total-elapsed 1h11m33s
epoch 1 batch 803/1,245 processed 25,696 batch-loss 11.6405 grad-norm 148.1256 epoch-elapsed 1h11m38s total-elapsed 1h11m38s
epoch 1 batch 804/1,245 processed 25,728 batch-loss 9.8608 grad-norm 107.0215 epoch-elapsed 1h11m41s total-elapsed 1h11m41s
epoch 1 batch 805/1,245 processed 25,760 batch-loss 8.2835 grad-norm 163.7454 epoch-elapsed 1h11m43s total-elapsed 1h11m43s
epoch 1 batch 806/1,245 processed 25,792 batch-loss 11.3994 grad-norm 101.8092 epoch-elapsed 1h11m46s total-elapsed 1h11m46s
epoch 1 batch 807/1,245 processed 25,824 batch-loss 9.8550 grad-norm 304.2078 epoch-elapsed 1h11m49s total-elapsed 1h11m49s
epoch 1 batch 808/1,245 processed 25,856 batch-loss 7.4891 grad-norm 138.9369 epoch-elapsed 1h11m52s total-elapsed 1h11m52s
epoch 1 batch 809/1,245 processed 25,888 batch-loss 7.7498 grad-norm 85.6025 epoch-elapsed 1h11m54s total-elapsed 1h11m54s
epoch 1 batch 810/1,245 processed 25,920 batch-loss 9.3191 grad-norm 128.7733 epoch-elapsed 1h11m57s total-elapsed 1h11m57s
epoch 1 batch 811/1,245 processed 25,952 batch-loss 10.8203 grad-norm 473.2210 epoch-elapsed 1h12m00s total-elapsed 1h12m00s
epoch 1 batch 812/1,245 processed 25,984 batch-loss 10.7135 grad-norm 100.9266 epoch-elapsed 1h12m03s total-elapsed 1h12m03s
epoch 1 batch 813/1,245 processed 26,016 batch-loss 9.1780 grad-norm 94.0275 epoch-elapsed 1h12m05s total-elapsed 1h12m05s
epoch 1 batch 814/1,245 processed 26,048 batch-loss 7.8341 grad-norm 57.0745 epoch-elapsed 1h12m08s total-elapsed 1h12m08s
epoch 1 batch 815/1,245 processed 26,080 batch-loss 11.5578 grad-norm 215.2595 epoch-elapsed 1h12m12s total-elapsed 1h12m12s
epoch 1 batch 816/1,245 processed 26,112 batch-loss 10.0208 grad-norm 127.3471 epoch-elapsed 1h12m15s total-elapsed 1h12m15s
epoch 1 batch 817/1,245 processed 26,144 batch-loss 6.9342 grad-norm 45.1907 epoch-elapsed 1h12m18s total-elapsed 1h12m18s
epoch 1 batch 818/1,245 processed 26,176 batch-loss 8.9944 grad-norm 93.2161 epoch-elapsed 1h12m21s total-elapsed 1h12m21s
epoch 1 batch 819/1,245 processed 26,208 batch-loss 8.6215 grad-norm 200.7254 epoch-elapsed 1h12m24s total-elapsed 1h12m24s
epoch 1 batch 820/1,245 processed 26,240 batch-loss 11.8550 grad-norm 170.1985 epoch-elapsed 1h12m28s total-elapsed 1h12m28s
epoch 1 batch 821/1,245 processed 26,272 batch-loss 9.6550 grad-norm 75.8378 epoch-elapsed 1h12m32s total-elapsed 1h12m32s
epoch 1 batch 822/1,245 processed 26,304 batch-loss 9.9155 grad-norm 105.8011 epoch-elapsed 1h12m36s total-elapsed 1h12m36s
epoch 1 batch 823/1,245 processed 26,336 batch-loss 11.0148 grad-norm 195.7517 epoch-elapsed 1h12m39s total-elapsed 1h12m39s
epoch 1 batch 824/1,245 processed 26,368 batch-loss 9.0198 grad-norm 90.0678 epoch-elapsed 1h12m42s total-elapsed 1h12m42s
epoch 1 batch 825/1,245 processed 26,400 batch-loss 9.1373 grad-norm 71.8527 epoch-elapsed 1h12m45s total-elapsed 1h12m45s
epoch 1 batch 826/1,245 processed 26,432 batch-loss 11.2851 grad-norm 141.0402 epoch-elapsed 1h12m49s total-elapsed 1h12m49s
epoch 1 batch 827/1,245 processed 26,464 batch-loss 9.5442 grad-norm 89.3812 epoch-elapsed 1h12m51s total-elapsed 1h12m51s
epoch 1 batch 828/1,245 processed 26,496 batch-loss 10.2126 grad-norm 70.1605 epoch-elapsed 1h12m55s total-elapsed 1h12m55s
epoch 1 batch 829/1,245 processed 26,528 batch-loss 6.3155 grad-norm 688.2753 epoch-elapsed 1h12m57s total-elapsed 1h12m57s
epoch 1 batch 830/1,245 processed 26,560 batch-loss 9.4704 grad-norm 59.9767 epoch-elapsed 1h13m01s total-elapsed 1h13m01s
epoch 1 batch 831/1,245 processed 26,592 batch-loss 9.8850 grad-norm 271.3098 epoch-elapsed 1h13m04s total-elapsed 1h13m04s
epoch 1 batch 832/1,245 processed 26,624 batch-loss 12.4201 grad-norm 441.5048 epoch-elapsed 1h13m08s total-elapsed 1h13m08s
epoch 1 batch 833/1,245 processed 26,656 batch-loss 10.5153 grad-norm 113.4248 epoch-elapsed 1h13m11s total-elapsed 1h13m11s
epoch 1 batch 834/1,245 processed 26,688 batch-loss 7.6653 grad-norm 82.3077 epoch-elapsed 1h13m14s total-elapsed 1h13m14s
epoch 1 batch 835/1,245 processed 26,720 batch-loss 9.0800 grad-norm 277.0056 epoch-elapsed 1h13m17s total-elapsed 1h13m17s
epoch 1 batch 836/1,245 processed 26,752 batch-loss 7.2707 grad-norm 82.6805 epoch-elapsed 1h13m20s total-elapsed 1h13m20s
epoch 1 batch 837/1,245 processed 26,784 batch-loss 6.5870 grad-norm 151.7377 epoch-elapsed 1h13m23s total-elapsed 1h13m23s
epoch 1 batch 838/1,245 processed 26,816 batch-loss 6.0462 grad-norm 275.8946 epoch-elapsed 1h13m25s total-elapsed 1h13m25s
epoch 1 batch 839/1,245 processed 26,848 batch-loss 9.3934 grad-norm 485.2977 epoch-elapsed 1h13m28s total-elapsed 1h13m28s
epoch 1 batch 840/1,245 processed 26,880 batch-loss 9.4808 grad-norm 156.9317 epoch-elapsed 1h13m32s total-elapsed 1h13m32s
epoch 1 batch 841/1,245 processed 26,912 batch-loss 9.5647 grad-norm 123.5107 epoch-elapsed 1h13m35s total-elapsed 1h13m35s
epoch 1 batch 842/1,245 processed 26,944 batch-loss 7.7046 grad-norm 1542.9206 epoch-elapsed 1h13m37s total-elapsed 1h13m37s
epoch 1 batch 843/1,245 processed 26,976 batch-loss 8.4890 grad-norm 193.5103 epoch-elapsed 1h13m40s total-elapsed 1h13m40s
epoch 1 batch 844/1,245 processed 27,008 batch-loss 7.5605 grad-norm 200.9250 epoch-elapsed 1h13m43s total-elapsed 1h13m43s
epoch 1 batch 845/1,245 processed 27,040 batch-loss 7.8266 grad-norm 72.2852 epoch-elapsed 1h13m47s total-elapsed 1h13m47s
epoch 1 batch 846/1,245 processed 27,072 batch-loss 7.0535 grad-norm 66.5913 epoch-elapsed 1h13m49s total-elapsed 1h13m49s
epoch 1 batch 847/1,245 processed 27,104 batch-loss 8.0868 grad-norm 153.9247 epoch-elapsed 1h13m52s total-elapsed 1h13m52s
epoch 1 batch 848/1,245 processed 27,136 batch-loss 7.2994 grad-norm 72.0369 epoch-elapsed 1h13m54s total-elapsed 1h13m54s
epoch 1 batch 849/1,245 processed 27,168 batch-loss 8.9521 grad-norm 127.4024 epoch-elapsed 1h13m58s total-elapsed 1h13m58s
epoch 1 batch 850/1,245 processed 27,200 batch-loss 7.9475 grad-norm 147.6449 epoch-elapsed 1h14m00s total-elapsed 1h14m00s
epoch 1 batch 851/1,245 processed 27,232 batch-loss 9.5446 grad-norm 178.1846 epoch-elapsed 1h14m04s total-elapsed 1h14m04s
epoch 1 batch 852/1,245 processed 27,264 batch-loss 11.4971 grad-norm 479.6415 epoch-elapsed 1h14m08s total-elapsed 1h14m08s
epoch 1 batch 853/1,245 processed 27,296 batch-loss 7.4764 grad-norm 65.7974 epoch-elapsed 1h14m10s total-elapsed 1h14m10s
epoch 1 batch 854/1,245 processed 27,328 batch-loss 9.0329 grad-norm 208.3692 epoch-elapsed 1h14m13s total-elapsed 1h14m13s
epoch 1 batch 855/1,245 processed 27,360 batch-loss 7.4932 grad-norm 556.5522 epoch-elapsed 1h14m16s total-elapsed 1h14m16s
epoch 1 batch 856/1,245 processed 27,392 batch-loss 8.4047 grad-norm 323.4123 epoch-elapsed 1h14m18s total-elapsed 1h14m18s
epoch 1 batch 857/1,245 processed 27,424 batch-loss 6.1520 grad-norm 109.8780 epoch-elapsed 1h14m21s total-elapsed 1h14m21s
epoch 1 batch 858/1,245 processed 27,456 batch-loss 5.9536 grad-norm 167.5242 epoch-elapsed 1h14m23s total-elapsed 1h14m23s
epoch 1 batch 859/1,245 processed 27,488 batch-loss 7.8793 grad-norm 116.5615 epoch-elapsed 1h14m26s total-elapsed 1h14m26s
epoch 1 batch 860/1,245 processed 27,520 batch-loss 9.9445 grad-norm 167.5372 epoch-elapsed 1h14m30s total-elapsed 1h14m30s
epoch 1 batch 861/1,245 processed 27,552 batch-loss 8.9324 grad-norm 232.8001 epoch-elapsed 1h14m33s total-elapsed 1h14m33s
epoch 1 batch 862/1,245 processed 27,584 batch-loss 7.3928 grad-norm 747.4311 epoch-elapsed 1h14m36s total-elapsed 1h14m36s
epoch 1 batch 863/1,245 processed 27,616 batch-loss 7.1988 grad-norm 67.6285 epoch-elapsed 1h14m39s total-elapsed 1h14m39s
epoch 1 batch 864/1,245 processed 27,648 batch-loss 9.4515 grad-norm 105.9974 epoch-elapsed 1h14m42s total-elapsed 1h14m42s
epoch 1 batch 865/1,245 processed 27,680 batch-loss 6.4033 grad-norm 75.9465 epoch-elapsed 1h14m44s total-elapsed 1h14m44s
epoch 1 batch 866/1,245 processed 27,712 batch-loss 10.1196 grad-norm 184.7228 epoch-elapsed 1h14m49s total-elapsed 1h14m49s
epoch 1 batch 867/1,245 processed 27,744 batch-loss 7.3111 grad-norm 314.4454 epoch-elapsed 1h14m51s total-elapsed 1h14m51s
epoch 1 batch 868/1,245 processed 27,776 batch-loss 9.5734 grad-norm 68.2805 epoch-elapsed 1h14m54s total-elapsed 1h14m54s
epoch 1 batch 869/1,245 processed 27,808 batch-loss 9.0265 grad-norm 64.8431 epoch-elapsed 1h14m57s total-elapsed 1h14m57s
epoch 1 batch 870/1,245 processed 27,840 batch-loss 8.1454 grad-norm 449.3139 epoch-elapsed 1h15m00s total-elapsed 1h15m00s
epoch 1 batch 871/1,245 processed 27,872 batch-loss 6.5438 grad-norm 349.2585 epoch-elapsed 1h15m03s total-elapsed 1h15m03s
epoch 1 batch 872/1,245 processed 27,904 batch-loss 8.4168 grad-norm 148.9991 epoch-elapsed 1h15m06s total-elapsed 1h15m06s
epoch 1 batch 873/1,245 processed 27,936 batch-loss 7.7793 grad-norm 96.8857 epoch-elapsed 1h15m10s total-elapsed 1h15m10s
epoch 1 batch 874/1,245 processed 27,968 batch-loss 5.2542 grad-norm 66.8900 epoch-elapsed 1h15m12s total-elapsed 1h15m12s
epoch 1 batch 875/1,245 processed 28,000 batch-loss 6.5993 grad-norm 65.4510 epoch-elapsed 1h15m15s total-elapsed 1h15m15s
epoch 1 batch 876/1,245 processed 28,032 batch-loss 10.4009 grad-norm 187.8823 epoch-elapsed 1h15m17s total-elapsed 1h15m17s
epoch 1 batch 877/1,245 processed 28,064 batch-loss 6.8730 grad-norm 78.3471 epoch-elapsed 1h15m20s total-elapsed 1h15m20s
epoch 1 batch 878/1,245 processed 28,096 batch-loss 4.6457 grad-norm 178.7830 epoch-elapsed 1h15m22s total-elapsed 1h15m22s
epoch 1 batch 879/1,245 processed 28,128 batch-loss 8.3282 grad-norm 482.0507 epoch-elapsed 1h15m26s total-elapsed 1h15m26s
epoch 1 batch 880/1,245 processed 28,160 batch-loss 11.0131 grad-norm 655.7450 epoch-elapsed 1h15m29s total-elapsed 1h15m29s
epoch 1 batch 881/1,245 processed 28,192 batch-loss 6.8739 grad-norm 373.3075 epoch-elapsed 1h15m31s total-elapsed 1h15m31s
epoch 1 batch 882/1,245 processed 28,224 batch-loss 7.7807 grad-norm 65.0186 epoch-elapsed 1h15m34s total-elapsed 1h15m34s
epoch 1 batch 883/1,245 processed 28,256 batch-loss 8.0809 grad-norm 138.5515 epoch-elapsed 1h15m36s total-elapsed 1h15m36s
epoch 1 batch 884/1,245 processed 28,288 batch-loss 8.1237 grad-norm 81.0850 epoch-elapsed 1h15m39s total-elapsed 1h15m39s
epoch 1 batch 885/1,245 processed 28,320 batch-loss 8.2765 grad-norm 314.0535 epoch-elapsed 1h15m42s total-elapsed 1h15m42s
epoch 1 batch 886/1,245 processed 28,352 batch-loss 8.9454 grad-norm 67.3528 epoch-elapsed 1h15m45s total-elapsed 1h15m45s
epoch 1 batch 887/1,245 processed 28,384 batch-loss 8.7190 grad-norm 148.6031 epoch-elapsed 1h15m48s total-elapsed 1h15m48s
epoch 1 batch 888/1,245 processed 28,416 batch-loss 8.9170 grad-norm 277.2584 epoch-elapsed 1h15m50s total-elapsed 1h15m51s
epoch 1 batch 889/1,245 processed 28,448 batch-loss 10.3886 grad-norm 112.0510 epoch-elapsed 1h15m54s total-elapsed 1h15m54s
epoch 1 batch 890/1,245 processed 28,480 batch-loss 6.3957 grad-norm 63.3892 epoch-elapsed 1h15m56s total-elapsed 1h15m56s
epoch 1 batch 891/1,245 processed 28,512 batch-loss 11.1207 grad-norm 608.2927 epoch-elapsed 1h15m59s total-elapsed 1h15m59s
epoch 1 batch 892/1,245 processed 28,544 batch-loss 7.3629 grad-norm 127.6920 epoch-elapsed 1h16m01s total-elapsed 1h16m01s
epoch 1 batch 893/1,245 processed 28,576 batch-loss 6.7639 grad-norm 70.5455 epoch-elapsed 1h16m03s total-elapsed 1h16m03s
epoch 1 batch 894/1,245 processed 28,608 batch-loss 7.5052 grad-norm 236.9089 epoch-elapsed 1h16m07s total-elapsed 1h16m07s
epoch 1 batch 895/1,245 processed 28,640 batch-loss 12.1824 grad-norm 169.3249 epoch-elapsed 1h16m11s total-elapsed 1h16m11s
epoch 1 batch 896/1,245 processed 28,672 batch-loss 6.4078 grad-norm 79.8106 epoch-elapsed 1h16m14s total-elapsed 1h16m14s
epoch 1 batch 897/1,245 processed 28,704 batch-loss 6.7242 grad-norm 147.0150 epoch-elapsed 1h16m16s total-elapsed 1h16m16s
epoch 1 batch 898/1,245 processed 28,736 batch-loss 9.8412 grad-norm 218.1156 epoch-elapsed 1h16m20s total-elapsed 1h16m20s
epoch 1 batch 899/1,245 processed 28,768 batch-loss 6.7421 grad-norm 75.9299 epoch-elapsed 1h16m22s total-elapsed 1h16m22s
epoch 1 batch 900/1,245 processed 28,800 batch-loss 8.5000 grad-norm 188.3259 epoch-elapsed 1h16m25s total-elapsed 1h16m25s
epoch 1 batch 901/1,245 processed 28,832 batch-loss 5.8588 grad-norm 130.1083 epoch-elapsed 1h16m27s total-elapsed 1h16m27s
epoch 1 batch 902/1,245 processed 28,864 batch-loss 9.0649 grad-norm 108.7996 epoch-elapsed 1h16m30s total-elapsed 1h16m30s
epoch 1 batch 903/1,245 processed 28,896 batch-loss 6.4942 grad-norm 73.4293 epoch-elapsed 1h16m32s total-elapsed 1h16m32s
epoch 1 batch 904/1,245 processed 28,928 batch-loss 9.6136 grad-norm 298.9169 epoch-elapsed 1h16m35s total-elapsed 1h16m35s
epoch 1 batch 905/1,245 processed 28,960 batch-loss 7.9805 grad-norm 95.6510 epoch-elapsed 1h16m37s total-elapsed 1h16m37s
epoch 1 batch 906/1,245 processed 28,992 batch-loss 9.7062 grad-norm 147.8636 epoch-elapsed 1h16m42s total-elapsed 1h16m42s
epoch 1 batch 907/1,245 processed 29,024 batch-loss 10.3484 grad-norm 257.7499 epoch-elapsed 1h16m45s total-elapsed 1h16m45s
epoch 1 batch 908/1,245 processed 29,056 batch-loss 7.9128 grad-norm 116.3461 epoch-elapsed 1h16m48s total-elapsed 1h16m48s
epoch 1 batch 909/1,245 processed 29,088 batch-loss 8.5132 grad-norm 54.3252 epoch-elapsed 1h16m52s total-elapsed 1h16m52s
epoch 1 batch 910/1,245 processed 29,120 batch-loss 8.3582 grad-norm 108.4137 epoch-elapsed 1h16m55s total-elapsed 1h16m55s
epoch 1 batch 911/1,245 processed 29,152 batch-loss 6.9909 grad-norm 614.5067 epoch-elapsed 1h16m58s total-elapsed 1h16m58s
epoch 1 batch 912/1,245 processed 29,184 batch-loss 10.5613 grad-norm 208.5330 epoch-elapsed 1h17m02s total-elapsed 1h17m02s
epoch 1 batch 913/1,245 processed 29,216 batch-loss 6.8189 grad-norm 367.6433 epoch-elapsed 1h17m05s total-elapsed 1h17m05s
epoch 1 batch 914/1,245 processed 29,248 batch-loss 9.5131 grad-norm 1024.6051 epoch-elapsed 1h17m08s total-elapsed 1h17m08s
epoch 1 batch 915/1,245 processed 29,280 batch-loss 6.6535 grad-norm 54.8754 epoch-elapsed 1h17m11s total-elapsed 1h17m11s
epoch 1 batch 916/1,245 processed 29,312 batch-loss 9.2037 grad-norm 379.1221 epoch-elapsed 1h17m14s total-elapsed 1h17m14s
epoch 1 batch 917/1,245 processed 29,344 batch-loss 6.9556 grad-norm 59.5739 epoch-elapsed 1h17m17s total-elapsed 1h17m17s
epoch 1 batch 918/1,245 processed 29,376 batch-loss 6.7403 grad-norm 64.9435 epoch-elapsed 1h17m20s total-elapsed 1h17m20s
epoch 1 batch 919/1,245 processed 29,408 batch-loss 5.8782 grad-norm 38.1230 epoch-elapsed 1h17m22s total-elapsed 1h17m22s
epoch 1 batch 920/1,245 processed 29,440 batch-loss 7.7466 grad-norm 161.1864 epoch-elapsed 1h17m25s total-elapsed 1h17m25s
epoch 1 batch 921/1,245 processed 29,472 batch-loss 8.3751 grad-norm 121.3106 epoch-elapsed 1h17m28s total-elapsed 1h17m28s
epoch 1 batch 922/1,245 processed 29,504 batch-loss 8.4539 grad-norm 154.5524 epoch-elapsed 1h17m30s total-elapsed 1h17m30s
epoch 1 batch 923/1,245 processed 29,536 batch-loss 7.0924 grad-norm 137.6711 epoch-elapsed 1h17m33s total-elapsed 1h17m33s
epoch 1 batch 924/1,245 processed 29,568 batch-loss 8.8751 grad-norm 257.9905 epoch-elapsed 1h17m36s total-elapsed 1h17m36s
epoch 1 batch 925/1,245 processed 29,600 batch-loss 5.6357 grad-norm 174.0848 epoch-elapsed 1h17m38s total-elapsed 1h17m38s
epoch 1 batch 926/1,245 processed 29,632 batch-loss 8.5193 grad-norm 77.6636 epoch-elapsed 1h17m41s total-elapsed 1h17m41s
epoch 1 batch 927/1,245 processed 29,664 batch-loss 8.7962 grad-norm 74.0005 epoch-elapsed 1h17m44s total-elapsed 1h17m44s
epoch 1 batch 928/1,245 processed 29,696 batch-loss 5.4214 grad-norm 59.1964 epoch-elapsed 1h17m46s total-elapsed 1h17m46s
epoch 1 batch 929/1,245 processed 29,728 batch-loss 5.0169 grad-norm 48.4418 epoch-elapsed 1h17m49s total-elapsed 1h17m49s
epoch 1 batch 930/1,245 processed 29,760 batch-loss 9.4169 grad-norm 730.3254 epoch-elapsed 1h17m52s total-elapsed 1h17m52s
epoch 1 batch 931/1,245 processed 29,792 batch-loss 7.7213 grad-norm 82.1882 epoch-elapsed 1h17m55s total-elapsed 1h17m55s
epoch 1 batch 932/1,245 processed 29,824 batch-loss 5.9449 grad-norm 360.6651 epoch-elapsed 1h17m57s total-elapsed 1h17m57s
epoch 1 batch 933/1,245 processed 29,856 batch-loss 9.2256 grad-norm 158.4530 epoch-elapsed 1h18m01s total-elapsed 1h18m01s
epoch 1 batch 934/1,245 processed 29,888 batch-loss 7.1283 grad-norm 67.3704 epoch-elapsed 1h18m04s total-elapsed 1h18m04s