FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  457.46         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.6024         	|167            	|  434.61         	|  95.004         	|
run_training_batch                 	|  0.38094        	|1000           	|  380.94         	|  83.273         	|
optimizer_step_with_closure_0      	|  0.37828        	|1000           	|  378.28         	|  82.691         	|
training_step_and_backward         	|  0.2982         	|1000           	|  298.2          	|  65.187         	|
model_forward                      	|  0.17256        	|1000           	|  172.56         	|  37.722         	|
training_step                      	|  0.17242        	|1000           	|  172.42         	|  37.689         	|
backward                           	|  0.1243         	|1000           	|  124.3          	|  27.172         	|
get_train_batch                    	|  0.014151       	|1167           	|  16.514         	|  3.6098         	|
fetch_next_train_batch             	|  0.014127       	|1167           	|  16.486         	|  3.6038         	|
zero_grad                          	|  0.0013132      	|1000           	|  1.3132         	|  0.28706        	|
on_train_batch_end                 	|  0.0012382      	|1000           	|  1.2382         	|  0.27066        	|
training_batch_to_device           	|  0.00031603     	|1000           	|  0.31603        	|  0.069082       	|
on_train_batch_start               	|  9.7769e-05     	|1000           	|  0.097769       	|  0.021372       	|
on_train_epoch_start               	|  0.00027863     	|167            	|  0.046532       	|  0.010172       	|
on_train_epoch_end                 	|  0.00026476     	|167            	|  0.044214       	|  0.0096652      	|
on_train_end                       	|  0.024927       	|1              	|  0.024927       	|  0.005449       	|
on_pretrain_routine_start          	|  0.022379       	|1              	|  0.022379       	|  0.004892       	|
on_batch_start                     	|  2.0998e-05     	|1000           	|  0.020998       	|  0.0045901      	|
on_before_zero_grad                	|  1.5212e-05     	|1000           	|  0.015212       	|  0.0033253      	|
on_batch_end                       	|  1.4491e-05     	|1000           	|  0.014491       	|  0.0031677      	|
on_before_optimizer_step           	|  1.2667e-05     	|1000           	|  0.012667       	|  0.0027691      	|
on_after_backward                  	|  1.1548e-05     	|1000           	|  0.011548       	|  0.0025243      	|
on_before_backward                 	|  1.0518e-05     	|1000           	|  0.010518       	|  0.0022991      	|
configure_optimizers               	|  0.0080399      	|1              	|  0.0080399      	|  0.0017575      	|
training_step_end                  	|  6.2164e-06     	|1000           	|  0.0062164      	|  0.0013589      	|
on_epoch_end                       	|  1.0611e-05     	|167            	|  0.001772       	|  0.00038735     	|
on_epoch_start                     	|  1.0545e-05     	|167            	|  0.0017611      	|  0.00038497     	|
on_train_start                     	|  0.00072378     	|1              	|  0.00072378     	|  0.00015822     	|
on_fit_end                         	|  2.2362e-05     	|1              	|  2.2362e-05     	|  4.8883e-06     	|
on_before_accelerator_backend_setup	|  1.9536e-05     	|1              	|  1.9536e-05     	|  4.2705e-06     	|
on_fit_start                       	|  1.7092e-05     	|1              	|  1.7092e-05     	|  3.7363e-06     	|
setup                              	|  1.583e-05      	|1              	|  1.583e-05      	|  3.4604e-06     	|
on_configure_sharded_model         	|  1.4077e-05     	|1              	|  1.4077e-05     	|  3.0772e-06     	|
teardown                           	|  1.11e-05       	|1              	|  1.11e-05       	|  2.4264e-06     	|
on_pretrain_routine_end            	|  9.818e-06      	|1              	|  9.818e-06      	|  2.1462e-06     	|
prepare_data                       	|  7.6239e-06     	|1              	|  7.6239e-06     	|  1.6666e-06     	|
configure_callbacks                	|  5.731e-06      	|1              	|  5.731e-06      	|  1.2528e-06     	|
on_val_dataloader                  	|  4.389e-06      	|1              	|  4.389e-06      	|  9.5942e-07     	|
configure_sharded_model            	|  4.138e-06      	|1              	|  4.138e-06      	|  9.0455e-07     	|
on_train_dataloader                	|  3.968e-06      	|1              	|  3.968e-06      	|  8.674e-07      	|
FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1558.1         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.1926         	|167            	|  1535.2         	|  98.526         	|
run_training_batch                 	|  0.18603        	|8000           	|  1488.2         	|  95.512         	|
training_step_and_backward         	|  0.17589        	|8000           	|  1407.1         	|  90.308         	|
model_forward                      	|  0.09382        	|8000           	|  750.56         	|  48.17          	|
training_step                      	|  0.093687       	|8000           	|  749.5          	|  48.102         	|
backward                           	|  0.081884       	|8000           	|  655.07         	|  42.042         	|
optimizer_step_with_closure_0      	|  0.23425        	|1000           	|  234.25         	|  15.034         	|
get_train_batch                    	|  0.0033435      	|8167           	|  27.306         	|  1.7525         	|
fetch_next_train_batch             	|  0.0033257      	|8167           	|  27.161         	|  1.7432         	|
on_train_batch_end                 	|  0.0010811      	|8000           	|  8.6485         	|  0.55506        	|
training_batch_to_device           	|  0.00022015     	|8000           	|  1.7612         	|  0.11303        	|
zero_grad                          	|  0.0013465      	|1000           	|  1.3465         	|  0.086414       	|
on_train_batch_start               	|  8.9635e-05     	|8000           	|  0.71708        	|  0.046022       	|
on_batch_start                     	|  1.5328e-05     	|8000           	|  0.12263        	|  0.0078702      	|
on_batch_end                       	|  1.2983e-05     	|8000           	|  0.10386        	|  0.0066657      	|
on_before_backward                 	|  1.2644e-05     	|8000           	|  0.10115        	|  0.0064919      	|
on_after_backward                  	|  1.2353e-05     	|8000           	|  0.09882        	|  0.0063422      	|
on_train_epoch_start               	|  0.00028389     	|167            	|  0.047409       	|  0.0030427      	|
training_step_end                  	|  5.716e-06      	|8000           	|  0.045728       	|  0.0029348      	|
on_train_epoch_end                 	|  0.00025165     	|167            	|  0.042026       	|  0.0026972      	|
on_train_end                       	|  0.025544       	|1              	|  0.025544       	|  0.0016394      	|
on_pretrain_routine_start          	|  0.02241        	|1              	|  0.02241        	|  0.0014382      	|
on_before_zero_grad                	|  1.5794e-05     	|1000           	|  0.015794       	|  0.0010137      	|
on_before_optimizer_step           	|  1.2005e-05     	|1000           	|  0.012005       	|  0.00077049     	|
configure_optimizers               	|  0.0085041      	|1              	|  0.0085041      	|  0.00054579     	|
on_epoch_start                     	|  1.1416e-05     	|167            	|  0.0019065      	|  0.00012236     	|
on_epoch_end                       	|  1.0156e-05     	|167            	|  0.0016961      	|  0.00010885     	|
on_train_start                     	|  0.00072419     	|1              	|  0.00072419     	|  4.6478e-05     	|
on_fit_end                         	|  2.5127e-05     	|1              	|  2.5127e-05     	|  1.6126e-06     	|
on_before_accelerator_backend_setup	|  1.8074e-05     	|1              	|  1.8074e-05     	|  1.16e-06       	|
on_fit_start                       	|  1.6521e-05     	|1              	|  1.6521e-05     	|  1.0603e-06     	|
on_configure_sharded_model         	|  1.3715e-05     	|1              	|  1.3715e-05     	|  8.8021e-07     	|
setup                              	|  1.2864e-05     	|1              	|  1.2864e-05     	|  8.256e-07      	|
on_pretrain_routine_end            	|  1.2664e-05     	|1              	|  1.2664e-05     	|  8.1276e-07     	|
teardown                           	|  1.2433e-05     	|1              	|  1.2433e-05     	|  7.9794e-07     	|
configure_callbacks                	|  6.8129e-06     	|1              	|  6.8129e-06     	|  4.3724e-07     	|
prepare_data                       	|  6.8031e-06     	|1              	|  6.8031e-06     	|  4.3662e-07     	|
on_val_dataloader                  	|  4.438e-06      	|1              	|  4.438e-06      	|  2.8483e-07     	|
configure_sharded_model            	|  4.3479e-06     	|1              	|  4.3479e-06     	|  2.7904e-07     	|
on_train_dataloader                	|  3.987e-06      	|1              	|  3.987e-06      	|  2.5588e-07     	|
