FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  463.17         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.636          	|167            	|  440.22         	|  95.045         	|
run_training_batch                 	|  0.38653        	|1000           	|  386.53         	|  83.454         	|
optimizer_step_with_closure_0      	|  0.38377        	|1000           	|  383.77         	|  82.857         	|
training_step_and_backward         	|  0.30399        	|1000           	|  303.99         	|  65.632         	|
model_forward                      	|  0.17555        	|1000           	|  175.55         	|  37.902         	|
training_step                      	|  0.1754         	|1000           	|  175.4          	|  37.871         	|
backward                           	|  0.12709        	|1000           	|  127.09         	|  27.44          	|
get_train_batch                    	|  0.01407        	|1167           	|  16.42          	|  3.5452         	|
fetch_next_train_batch             	|  0.014046       	|1167           	|  16.392         	|  3.5391         	|
zero_grad                          	|  0.0013162      	|1000           	|  1.3162         	|  0.28418        	|
on_train_batch_end                 	|  0.0012482      	|1000           	|  1.2482         	|  0.26949        	|
training_batch_to_device           	|  0.00031643     	|1000           	|  0.31643        	|  0.06832        	|
on_train_batch_start               	|  9.5734e-05     	|1000           	|  0.095734       	|  0.020669       	|
on_train_epoch_start               	|  0.00028296     	|167            	|  0.047255       	|  0.010203       	|
on_train_epoch_end                 	|  0.00025924     	|167            	|  0.043294       	|  0.0093473      	|
on_train_end                       	|  0.024701       	|1              	|  0.024701       	|  0.0053332      	|
on_pretrain_routine_start          	|  0.02235        	|1              	|  0.02235        	|  0.0048256      	|
on_batch_start                     	|  2.0968e-05     	|1000           	|  0.020968       	|  0.004527       	|
on_before_zero_grad                	|  1.5319e-05     	|1000           	|  0.015319       	|  0.0033075      	|
on_batch_end                       	|  1.3806e-05     	|1000           	|  0.013806       	|  0.0029808      	|
on_before_backward                 	|  1.1955e-05     	|1000           	|  0.011955       	|  0.0025811      	|
on_before_optimizer_step           	|  1.1497e-05     	|1000           	|  0.011497       	|  0.0024823      	|
on_after_backward                  	|  1.1437e-05     	|1000           	|  0.011437       	|  0.0024693      	|
configure_optimizers               	|  0.0086829      	|1              	|  0.0086829      	|  0.0018747      	|
training_step_end                  	|  6.6664e-06     	|1000           	|  0.0066664      	|  0.0014393      	|
on_epoch_start                     	|  1.1042e-05     	|167            	|  0.0018441      	|  0.00039814     	|
on_epoch_end                       	|  1.0768e-05     	|167            	|  0.0017983      	|  0.00038827     	|
on_train_start                     	|  0.00072588     	|1              	|  0.00072588     	|  0.00015672     	|
on_fit_end                         	|  2.3985e-05     	|1              	|  2.3985e-05     	|  5.1785e-06     	|
on_before_accelerator_backend_setup	|  2.0969e-05     	|1              	|  2.0969e-05     	|  4.5273e-06     	|
on_fit_start                       	|  1.7132e-05     	|1              	|  1.7132e-05     	|  3.6989e-06     	|
setup                              	|  1.5679e-05     	|1              	|  1.5679e-05     	|  3.3852e-06     	|
on_configure_sharded_model         	|  1.3886e-05     	|1              	|  1.3886e-05     	|  2.9981e-06     	|
on_pretrain_routine_end            	|  1.043e-05      	|1              	|  1.043e-05      	|  2.2519e-06     	|
teardown                           	|  1.0199e-05     	|1              	|  1.0199e-05     	|  2.202e-06      	|
prepare_data                       	|  9.0069e-06     	|1              	|  9.0069e-06     	|  1.9446e-06     	|
configure_callbacks                	|  7.1441e-06     	|1              	|  7.1441e-06     	|  1.5424e-06     	|
on_val_dataloader                  	|  4.8389e-06     	|1              	|  4.8389e-06     	|  1.0447e-06     	|
on_train_dataloader                	|  4.2591e-06     	|1              	|  4.2591e-06     	|  9.1955e-07     	|
configure_sharded_model            	|  4.1979e-06     	|1              	|  4.1979e-06     	|  9.0636e-07     	|
FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1555.4         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.1764         	|167            	|  1532.5         	|  98.528         	|
run_training_batch                 	|  0.18566        	|8000           	|  1485.3         	|  95.494         	|
training_step_and_backward         	|  0.17564        	|8000           	|  1405.1         	|  90.341         	|
model_forward                      	|  0.094062       	|8000           	|  752.5          	|  48.381         	|
training_step                      	|  0.093927       	|8000           	|  751.42         	|  48.312         	|
backward                           	|  0.081398       	|8000           	|  651.18         	|  41.867         	|
optimizer_step_with_closure_0      	|  0.23296        	|1000           	|  232.96         	|  14.978         	|
get_train_batch                    	|  0.0033534      	|8167           	|  27.387         	|  1.7608         	|
fetch_next_train_batch             	|  0.0033355      	|8167           	|  27.241         	|  1.7514         	|
on_train_batch_end                 	|  0.0010938      	|8000           	|  8.7508         	|  0.56262        	|
training_batch_to_device           	|  0.00021882     	|8000           	|  1.7505         	|  0.11255        	|
zero_grad                          	|  0.0012914      	|1000           	|  1.2914         	|  0.08303        	|
on_train_batch_start               	|  9.2084e-05     	|8000           	|  0.73667        	|  0.047363       	|
on_batch_start                     	|  1.5589e-05     	|8000           	|  0.12471        	|  0.0080183      	|
on_before_backward                 	|  1.3505e-05     	|8000           	|  0.10804        	|  0.0069463      	|
on_batch_end                       	|  1.305e-05      	|8000           	|  0.1044         	|  0.0067125      	|
on_after_backward                  	|  1.194e-05      	|8000           	|  0.09552        	|  0.0061414      	|
training_step_end                  	|  6.2779e-06     	|8000           	|  0.050223       	|  0.003229       	|
on_train_epoch_start               	|  0.00028302     	|167            	|  0.047265       	|  0.0030389      	|
on_train_epoch_end                 	|  0.0002502      	|167            	|  0.041784       	|  0.0026865      	|
on_train_end                       	|  0.026389       	|1              	|  0.026389       	|  0.0016966      	|
on_pretrain_routine_start          	|  0.022556       	|1              	|  0.022556       	|  0.0014502      	|
on_before_zero_grad                	|  1.6477e-05     	|1000           	|  0.016477       	|  0.0010594      	|
on_before_optimizer_step           	|  1.3777e-05     	|1000           	|  0.013777       	|  0.00088579     	|
configure_optimizers               	|  0.0081107      	|1              	|  0.0081107      	|  0.00052147     	|
on_epoch_end                       	|  1.0971e-05     	|167            	|  0.0018321      	|  0.00011779     	|
on_epoch_start                     	|  1.0662e-05     	|167            	|  0.0017805      	|  0.00011447     	|
on_train_start                     	|  0.00069518     	|1              	|  0.00069518     	|  4.4696e-05     	|
on_fit_end                         	|  2.1832e-05     	|1              	|  2.1832e-05     	|  1.4037e-06     	|
on_before_accelerator_backend_setup	|  2.0669e-05     	|1              	|  2.0669e-05     	|  1.3289e-06     	|
on_fit_start                       	|  1.7062e-05     	|1              	|  1.7062e-05     	|  1.097e-06      	|
setup                              	|  1.4337e-05     	|1              	|  1.4337e-05     	|  9.2178e-07     	|
on_configure_sharded_model         	|  1.3836e-05     	|1              	|  1.3836e-05     	|  8.8957e-07     	|
teardown                           	|  1.073e-05      	|1              	|  1.073e-05      	|  6.8987e-07     	|
on_pretrain_routine_end            	|  1.0339e-05     	|1              	|  1.0339e-05     	|  6.6474e-07     	|
prepare_data                       	|  8.035e-06      	|1              	|  8.035e-06      	|  5.166e-07      	|
configure_callbacks                	|  6.9828e-06     	|1              	|  6.9828e-06     	|  4.4895e-07     	|
configure_sharded_model            	|  4.8792e-06     	|1              	|  4.8792e-06     	|  3.137e-07      	|
on_val_dataloader                  	|  4.879e-06      	|1              	|  4.879e-06      	|  3.1369e-07     	|
on_train_dataloader                	|  3.9679e-06     	|1              	|  3.9679e-06     	|  2.5511e-07     	|
