FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  468.57         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.6657         	|167            	|  445.17         	|  95.005         	|
run_training_batch                 	|  0.38829        	|1000           	|  388.29         	|  82.867         	|
optimizer_step_with_closure_0      	|  0.38556        	|1000           	|  385.56         	|  82.283         	|
training_step_and_backward         	|  0.30242        	|1000           	|  302.42         	|  64.541         	|
model_forward                      	|  0.17466        	|1000           	|  174.66         	|  37.276         	|
training_step                      	|  0.17451        	|1000           	|  174.51         	|  37.244         	|
backward                           	|  0.12639        	|1000           	|  126.39         	|  26.973         	|
get_train_batch                    	|  0.014831       	|1167           	|  17.307         	|  3.6937         	|
fetch_next_train_batch             	|  0.014807       	|1167           	|  17.28          	|  3.6877         	|
zero_grad                          	|  0.0013438      	|1000           	|  1.3438         	|  0.28679        	|
on_train_batch_end                 	|  0.001257       	|1000           	|  1.257          	|  0.26826        	|
training_batch_to_device           	|  0.0003152      	|1000           	|  0.3152         	|  0.067268       	|
on_train_batch_start               	|  9.6694e-05     	|1000           	|  0.096694       	|  0.020636       	|
on_train_epoch_start               	|  0.00028994     	|167            	|  0.048419       	|  0.010333       	|
on_train_epoch_end                 	|  0.00027565     	|167            	|  0.046034       	|  0.0098242      	|
on_train_end                       	|  0.024661       	|1              	|  0.024661       	|  0.0052631      	|
on_pretrain_routine_start          	|  0.022839       	|1              	|  0.022839       	|  0.0048743      	|
on_batch_start                     	|  2.1606e-05     	|1000           	|  0.021606       	|  0.0046111      	|
on_before_zero_grad                	|  1.5737e-05     	|1000           	|  0.015737       	|  0.0033585      	|
on_batch_end                       	|  1.391e-05      	|1000           	|  0.01391        	|  0.0029687      	|
on_after_backward                  	|  1.3562e-05     	|1000           	|  0.013562       	|  0.0028942      	|
on_before_optimizer_step           	|  1.2186e-05     	|1000           	|  0.012186       	|  0.0026007      	|
on_before_backward                 	|  1.0462e-05     	|1000           	|  0.010462       	|  0.0022328      	|
configure_optimizers               	|  0.0085319      	|1              	|  0.0085319      	|  0.0018208      	|
training_step_end                  	|  6.6595e-06     	|1000           	|  0.0066595      	|  0.0014212      	|
on_epoch_start                     	|  1.0631e-05     	|167            	|  0.0017755      	|  0.00037891     	|
on_epoch_end                       	|  1.0534e-05     	|167            	|  0.0017591      	|  0.00037542     	|
on_train_start                     	|  0.00077146     	|1              	|  0.00077146     	|  0.00016464     	|
on_before_accelerator_backend_setup	|  2.2381e-05     	|1              	|  2.2381e-05     	|  4.7764e-06     	|
on_fit_end                         	|  2.168e-05      	|1              	|  2.168e-05      	|  4.6268e-06     	|
on_fit_start                       	|  1.9236e-05     	|1              	|  1.9236e-05     	|  4.1052e-06     	|
setup                              	|  1.586e-05      	|1              	|  1.586e-05      	|  3.3847e-06     	|
on_configure_sharded_model         	|  1.3566e-05     	|1              	|  1.3566e-05     	|  2.8952e-06     	|
teardown                           	|  1.064e-05      	|1              	|  1.064e-05      	|  2.2707e-06     	|
on_pretrain_routine_end            	|  1.061e-05      	|1              	|  1.061e-05      	|  2.2644e-06     	|
prepare_data                       	|  8.506e-06      	|1              	|  8.506e-06      	|  1.8153e-06     	|
configure_callbacks                	|  7.033e-06      	|1              	|  7.033e-06      	|  1.5009e-06     	|
on_val_dataloader                  	|  4.739e-06      	|1              	|  4.739e-06      	|  1.0114e-06     	|
on_train_dataloader                	|  4.549e-06      	|1              	|  4.549e-06      	|  9.7083e-07     	|
configure_sharded_model            	|  4.4981e-06     	|1              	|  4.4981e-06     	|  9.5995e-07     	|
FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1584.4         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.3242         	|167            	|  1557.1         	|  98.28          	|
run_training_batch                 	|  0.18873        	|8000           	|  1509.9         	|  95.296         	|
training_step_and_backward         	|  0.17837        	|8000           	|  1427.0         	|  90.064         	|
model_forward                      	|  0.094237       	|8000           	|  753.9          	|  47.583         	|
training_step                      	|  0.094102       	|8000           	|  752.81         	|  47.514         	|
backward                           	|  0.08395        	|8000           	|  671.6          	|  42.388         	|
optimizer_step_with_closure_0      	|  0.23854        	|1000           	|  238.54         	|  15.055         	|
get_train_batch                    	|  0.0033615      	|8167           	|  27.453         	|  1.7327         	|
fetch_next_train_batch             	|  0.0033439      	|8167           	|  27.309         	|  1.7237         	|
on_train_batch_end                 	|  0.0010775      	|8000           	|  8.6199         	|  0.54405        	|
training_batch_to_device           	|  0.00021714     	|8000           	|  1.7371         	|  0.10964        	|
zero_grad                          	|  0.0013227      	|1000           	|  1.3227         	|  0.083483       	|
on_train_batch_start               	|  9.1569e-05     	|8000           	|  0.73255        	|  0.046236       	|
on_batch_start                     	|  1.493e-05      	|8000           	|  0.11944        	|  0.0075385      	|
on_batch_end                       	|  1.3227e-05     	|8000           	|  0.10581        	|  0.0066785      	|
on_before_backward                 	|  1.2966e-05     	|8000           	|  0.10373        	|  0.0065469      	|
on_after_backward                  	|  1.136e-05      	|8000           	|  0.090884       	|  0.0057362      	|
training_step_end                  	|  6.0564e-06     	|8000           	|  0.048451       	|  0.003058       	|
on_train_epoch_start               	|  0.00028303     	|167            	|  0.047266       	|  0.0029832      	|
on_train_epoch_end                 	|  0.00025889     	|167            	|  0.043235       	|  0.0027288      	|
on_train_end                       	|  0.02599        	|1              	|  0.02599        	|  0.0016404      	|
on_pretrain_routine_start          	|  0.022534       	|1              	|  0.022534       	|  0.0014222      	|
on_before_zero_grad                	|  1.5746e-05     	|1000           	|  0.015746       	|  0.00099381     	|
on_before_optimizer_step           	|  1.358e-05      	|1000           	|  0.01358        	|  0.00085711     	|
configure_optimizers               	|  0.0083933      	|1              	|  0.0083933      	|  0.00052975     	|
on_epoch_start                     	|  1.078e-05      	|167            	|  0.0018002      	|  0.00011362     	|
on_epoch_end                       	|  1.0417e-05     	|167            	|  0.0017396      	|  0.0001098      	|
on_train_start                     	|  0.00072522     	|1              	|  0.00072522     	|  4.5773e-05     	|
on_before_accelerator_backend_setup	|  2.1941e-05     	|1              	|  2.1941e-05     	|  1.3848e-06     	|
on_fit_end                         	|  2.123e-05      	|1              	|  2.123e-05      	|  1.3399e-06     	|
on_fit_start                       	|  1.7422e-05     	|1              	|  1.7422e-05     	|  1.0996e-06     	|
on_configure_sharded_model         	|  1.3686e-05     	|1              	|  1.3686e-05     	|  8.638e-07      	|
setup                              	|  1.3144e-05     	|1              	|  1.3144e-05     	|  8.2959e-07     	|
on_pretrain_routine_end            	|  1.2093e-05     	|1              	|  1.2093e-05     	|  7.6326e-07     	|
teardown                           	|  9.9889e-06     	|1              	|  9.9889e-06     	|  6.3046e-07     	|
prepare_data                       	|  7.7151e-06     	|1              	|  7.7151e-06     	|  4.8694e-07     	|
configure_callbacks                	|  7.414e-06      	|1              	|  7.414e-06      	|  4.6794e-07     	|
configure_sharded_model            	|  5.149e-06      	|1              	|  5.149e-06      	|  3.2499e-07     	|
on_val_dataloader                  	|  4.3381e-06     	|1              	|  4.3381e-06     	|  2.738e-07      	|
on_train_dataloader                	|  4.068e-06      	|1              	|  4.068e-06      	|  2.5676e-07     	|
