FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  381.95         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.6831         	|63             	|  358.03         	|  93.739         	|
run_training_batch                 	|  0.33133        	|1000           	|  331.33         	|  86.747         	|
optimizer_step_with_closure_0      	|  0.32866        	|1000           	|  328.66         	|  86.048         	|
training_step_and_backward         	|  0.24601        	|1000           	|  246.01         	|  64.41          	|
model_forward                      	|  0.15322        	|1000           	|  153.22         	|  40.116         	|
training_step                      	|  0.15309        	|1000           	|  153.09         	|  40.08          	|
backward                           	|  0.091479       	|1000           	|  91.479         	|  23.951         	|
get_train_batch                    	|  0.0080987      	|1063           	|  8.6089         	|  2.2539         	|
fetch_next_train_batch             	|  0.0080785      	|1063           	|  8.5874         	|  2.2483         	|
zero_grad                          	|  0.0012876      	|1000           	|  1.2876         	|  0.33712        	|
on_train_batch_end                 	|  0.0012041      	|1000           	|  1.2041         	|  0.31525        	|
training_batch_to_device           	|  0.00022779     	|1000           	|  0.22779        	|  0.05964        	|
on_train_batch_start               	|  9.1913e-05     	|1000           	|  0.091913       	|  0.024064       	|
on_train_end                       	|  0.025653       	|1              	|  0.025653       	|  0.0067163      	|
on_pretrain_routine_start          	|  0.022552       	|1              	|  0.022552       	|  0.0059044      	|
on_train_epoch_start               	|  0.00030573     	|63             	|  0.019261       	|  0.0050429      	|
on_train_epoch_end                 	|  0.00028056     	|63             	|  0.017675       	|  0.0046276      	|
on_batch_start                     	|  1.5801e-05     	|1000           	|  0.015801       	|  0.0041369      	|
on_before_zero_grad                	|  1.3543e-05     	|1000           	|  0.013543       	|  0.0035458      	|
on_batch_end                       	|  1.2942e-05     	|1000           	|  0.012942       	|  0.0033884      	|
on_after_backward                  	|  1.1345e-05     	|1000           	|  0.011345       	|  0.0029704      	|
on_before_optimizer_step           	|  1.0573e-05     	|1000           	|  0.010573       	|  0.0027681      	|
on_before_backward                 	|  1.0299e-05     	|1000           	|  0.010299       	|  0.0026965      	|
configure_optimizers               	|  0.0081272      	|1              	|  0.0081272      	|  0.0021278      	|
training_step_end                  	|  6.024e-06      	|1000           	|  0.006024       	|  0.0015772      	|
on_train_start                     	|  0.00072426     	|1              	|  0.00072426     	|  0.00018962     	|
on_epoch_start                     	|  1.0469e-05     	|63             	|  0.00065953     	|  0.00017267     	|
on_epoch_end                       	|  1.0112e-05     	|63             	|  0.00063704     	|  0.00016679     	|
on_fit_end                         	|  2.2071e-05     	|1              	|  2.2071e-05     	|  5.7785e-06     	|
on_before_accelerator_backend_setup	|  1.8745e-05     	|1              	|  1.8745e-05     	|  4.9077e-06     	|
on_fit_start                       	|  1.559e-05      	|1              	|  1.559e-05      	|  4.0817e-06     	|
setup                              	|  1.3415e-05     	|1              	|  1.3415e-05     	|  3.5122e-06     	|
on_configure_sharded_model         	|  1.3205e-05     	|1              	|  1.3205e-05     	|  3.4573e-06     	|
teardown                           	|  1.047e-05      	|1              	|  1.047e-05      	|  2.7412e-06     	|
on_pretrain_routine_end            	|  9.789e-06      	|1              	|  9.789e-06      	|  2.5629e-06     	|
prepare_data                       	|  7.2641e-06     	|1              	|  7.2641e-06     	|  1.9018e-06     	|
configure_callbacks                	|  4.889e-06      	|1              	|  4.889e-06      	|  1.28e-06       	|
on_val_dataloader                  	|  4.829e-06      	|1              	|  4.829e-06      	|  1.2643e-06     	|
configure_sharded_model            	|  4.278e-06      	|1              	|  4.278e-06      	|  1.1201e-06     	|
on_train_dataloader                	|  4.127e-06      	|1              	|  4.127e-06      	|  1.0805e-06     	|
FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1562.7         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.438         	|63             	|  1539.6         	|  98.525         	|
run_training_batch                 	|  0.18806        	|8000           	|  1504.5         	|  96.276         	|
training_step_and_backward         	|  0.17778        	|8000           	|  1422.3         	|  91.014         	|
model_forward                      	|  0.092541       	|8000           	|  740.32         	|  47.375         	|
training_step                      	|  0.092408       	|8000           	|  739.26         	|  47.307         	|
backward                           	|  0.085056       	|8000           	|  680.45         	|  43.544         	|
optimizer_step_with_closure_0      	|  0.23666        	|1000           	|  236.66         	|  15.145         	|
get_train_batch                    	|  0.0022612      	|8063           	|  18.232         	|  1.1667         	|
fetch_next_train_batch             	|  0.0022445      	|8063           	|  18.097         	|  1.1581         	|
on_train_batch_end                 	|  0.0010793      	|8000           	|  8.6344         	|  0.55254        	|
training_batch_to_device           	|  0.00021568     	|8000           	|  1.7255         	|  0.11042        	|
zero_grad                          	|  0.0013301      	|1000           	|  1.3301         	|  0.085114       	|
on_train_batch_start               	|  9.2671e-05     	|8000           	|  0.74136        	|  0.047442       	|
on_batch_start                     	|  1.6838e-05     	|8000           	|  0.1347         	|  0.0086198      	|
on_batch_end                       	|  1.5757e-05     	|8000           	|  0.12606        	|  0.0080668      	|
on_before_backward                 	|  1.5265e-05     	|8000           	|  0.12212        	|  0.007815       	|
on_after_backward                  	|  1.385e-05      	|8000           	|  0.1108         	|  0.0070905      	|
training_step_end                  	|  5.7851e-06     	|8000           	|  0.046281       	|  0.0029617      	|
on_train_end                       	|  0.02538        	|1              	|  0.02538        	|  0.0016241      	|
on_pretrain_routine_start          	|  0.023022       	|1              	|  0.023022       	|  0.0014732      	|
on_train_epoch_start               	|  0.00029403     	|63             	|  0.018524       	|  0.0011854      	|
on_train_epoch_end                 	|  0.00028358     	|63             	|  0.017865       	|  0.0011433      	|
on_before_zero_grad                	|  1.6902e-05     	|1000           	|  0.016902       	|  0.0010816      	|
on_before_optimizer_step           	|  1.3738e-05     	|1000           	|  0.013738       	|  0.00087913     	|
configure_optimizers               	|  0.0081893      	|1              	|  0.0081893      	|  0.00052406     	|
on_epoch_start                     	|  1.2731e-05     	|63             	|  0.00080204     	|  5.1324e-05     	|
on_epoch_end                       	|  1.2495e-05     	|63             	|  0.00078717     	|  5.0373e-05     	|
on_train_start                     	|  0.00074195     	|1              	|  0.00074195     	|  4.7479e-05     	|
on_fit_end                         	|  2.3324e-05     	|1              	|  2.3324e-05     	|  1.4926e-06     	|
on_before_accelerator_backend_setup	|  2.2723e-05     	|1              	|  2.2723e-05     	|  1.4541e-06     	|
on_fit_start                       	|  1.9917e-05     	|1              	|  1.9917e-05     	|  1.2745e-06     	|
on_configure_sharded_model         	|  1.5339e-05     	|1              	|  1.5339e-05     	|  9.8158e-07     	|
setup                              	|  1.4898e-05     	|1              	|  1.4898e-05     	|  9.5336e-07     	|
on_pretrain_routine_end            	|  1.4617e-05     	|1              	|  1.4617e-05     	|  9.3537e-07     	|
teardown                           	|  1.2203e-05     	|1              	|  1.2203e-05     	|  7.8089e-07     	|
prepare_data                       	|  7.574e-06      	|1              	|  7.574e-06      	|  4.8468e-07     	|
configure_callbacks                	|  6.292e-06      	|1              	|  6.292e-06      	|  4.0264e-07     	|
on_val_dataloader                  	|  4.5691e-06     	|1              	|  4.5691e-06     	|  2.9239e-07     	|
on_train_dataloader                	|  4.5381e-06     	|1              	|  4.5381e-06     	|  2.9041e-07     	|
configure_sharded_model            	|  4.5379e-06     	|1              	|  4.5379e-06     	|  2.9039e-07     	|
