FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  385.83         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.7477         	|63             	|  362.11         	|  93.85          	|
run_training_batch                 	|  0.33539        	|1000           	|  335.39         	|  86.927         	|
optimizer_step_with_closure_0      	|  0.33283        	|1000           	|  332.83         	|  86.263         	|
training_step_and_backward         	|  0.24858        	|1000           	|  248.58         	|  64.426         	|
model_forward                      	|  0.15458        	|1000           	|  154.58         	|  40.063         	|
training_step                      	|  0.15444        	|1000           	|  154.44         	|  40.027         	|
backward                           	|  0.092668       	|1000           	|  92.668         	|  24.018         	|
get_train_batch                    	|  0.0081321      	|1063           	|  8.6445         	|  2.2405         	|
fetch_next_train_batch             	|  0.0081135      	|1063           	|  8.6247         	|  2.2353         	|
zero_grad                          	|  0.0013101      	|1000           	|  1.3101         	|  0.33954        	|
on_train_batch_end                 	|  0.0011792      	|1000           	|  1.1792         	|  0.30562        	|
training_batch_to_device           	|  0.00022517     	|1000           	|  0.22517        	|  0.058361       	|
on_train_batch_start               	|  9.4063e-05     	|1000           	|  0.094063       	|  0.024379       	|
on_train_end                       	|  0.025655       	|1              	|  0.025655       	|  0.0066491      	|
on_pretrain_routine_start          	|  0.022736       	|1              	|  0.022736       	|  0.0058927      	|
on_train_epoch_start               	|  0.00030499     	|63             	|  0.019214       	|  0.0049799      	|
on_train_epoch_end                 	|  0.00028987     	|63             	|  0.018262       	|  0.0047331      	|
on_batch_start                     	|  1.6342e-05     	|1000           	|  0.016342       	|  0.0042356      	|
on_batch_end                       	|  1.3704e-05     	|1000           	|  0.013704       	|  0.0035519      	|
on_before_zero_grad                	|  1.3512e-05     	|1000           	|  0.013512       	|  0.003502       	|
on_after_backward                  	|  1.1416e-05     	|1000           	|  0.011416       	|  0.0029587      	|
on_before_backward                 	|  1.1121e-05     	|1000           	|  0.011121       	|  0.0028822      	|
on_before_optimizer_step           	|  1.0522e-05     	|1000           	|  0.010522       	|  0.0027271      	|
configure_optimizers               	|  0.0082292      	|1              	|  0.0082292      	|  0.0021328      	|
training_step_end                  	|  5.9225e-06     	|1000           	|  0.0059225      	|  0.001535       	|
on_train_start                     	|  0.00073773     	|1              	|  0.00073773     	|  0.0001912      	|
on_epoch_start                     	|  1.066e-05      	|63             	|  0.00067161     	|  0.00017407     	|
on_epoch_end                       	|  1.0356e-05     	|63             	|  0.00065241     	|  0.00016909     	|
on_fit_end                         	|  2.139e-05      	|1              	|  2.139e-05      	|  5.5439e-06     	|
on_before_accelerator_backend_setup	|  2.0829e-05     	|1              	|  2.0829e-05     	|  5.3985e-06     	|
on_fit_start                       	|  1.6311e-05     	|1              	|  1.6311e-05     	|  4.2275e-06     	|
setup                              	|  1.4267e-05     	|1              	|  1.4267e-05     	|  3.6977e-06     	|
on_pretrain_routine_end            	|  1.4157e-05     	|1              	|  1.4157e-05     	|  3.6692e-06     	|
on_configure_sharded_model         	|  1.3325e-05     	|1              	|  1.3325e-05     	|  3.4536e-06     	|
teardown                           	|  1.085e-05      	|1              	|  1.085e-05      	|  2.8121e-06     	|
prepare_data                       	|  8.105e-06      	|1              	|  8.105e-06      	|  2.1006e-06     	|
configure_callbacks                	|  4.9301e-06     	|1              	|  4.9301e-06     	|  1.2778e-06     	|
configure_sharded_model            	|  4.6289e-06     	|1              	|  4.6289e-06     	|  1.1997e-06     	|
on_train_dataloader                	|  4.3879e-06     	|1              	|  4.3879e-06     	|  1.1373e-06     	|
on_val_dataloader                  	|  4.278e-06      	|1              	|  4.278e-06      	|  1.1088e-06     	|
FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1547.4         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.197         	|63             	|  1524.4         	|  98.516         	|
run_training_batch                 	|  0.18619        	|8000           	|  1489.5         	|  96.261         	|
training_step_and_backward         	|  0.17602        	|8000           	|  1408.2         	|  91.003         	|
model_forward                      	|  0.092195       	|8000           	|  737.56         	|  47.665         	|
training_step                      	|  0.092064       	|8000           	|  736.51         	|  47.597         	|
backward                           	|  0.083643       	|8000           	|  669.15         	|  43.244         	|
optimizer_step_with_closure_0      	|  0.2339         	|1000           	|  233.9          	|  15.116         	|
get_train_batch                    	|  0.0022549      	|8063           	|  18.181         	|  1.175          	|
fetch_next_train_batch             	|  0.0022385      	|8063           	|  18.049         	|  1.1664         	|
on_train_batch_end                 	|  0.0010788      	|8000           	|  8.6301         	|  0.55772        	|
training_batch_to_device           	|  0.00021122     	|8000           	|  1.6898         	|  0.1092         	|
zero_grad                          	|  0.0013202      	|1000           	|  1.3202         	|  0.085316       	|
on_train_batch_start               	|  8.9638e-05     	|8000           	|  0.7171         	|  0.046343       	|
on_batch_start                     	|  1.4344e-05     	|8000           	|  0.11475        	|  0.0074157      	|
on_batch_end                       	|  1.3165e-05     	|8000           	|  0.10532        	|  0.0068062      	|
on_before_backward                 	|  1.2588e-05     	|8000           	|  0.1007         	|  0.0065079      	|
on_after_backward                  	|  1.1586e-05     	|8000           	|  0.092688       	|  0.00599        	|
training_step_end                  	|  5.9807e-06     	|8000           	|  0.047846       	|  0.003092       	|
on_train_end                       	|  0.025971       	|1              	|  0.025971       	|  0.0016784      	|
on_pretrain_routine_start          	|  0.022093       	|1              	|  0.022093       	|  0.0014278      	|
on_train_epoch_start               	|  0.00030965     	|63             	|  0.019508       	|  0.0012607      	|
on_train_epoch_end                 	|  0.00027804     	|63             	|  0.017517       	|  0.001132       	|
on_before_zero_grad                	|  1.4834e-05     	|1000           	|  0.014834       	|  0.00095863     	|
on_before_optimizer_step           	|  1.2039e-05     	|1000           	|  0.012039       	|  0.00077804     	|
configure_optimizers               	|  0.0078151      	|1              	|  0.0078151      	|  0.00050505     	|
on_train_start                     	|  0.00070094     	|1              	|  0.00070094     	|  4.5298e-05     	|
on_epoch_start                     	|  1.0494e-05     	|63             	|  0.00066111     	|  4.2724e-05     	|
on_epoch_end                       	|  1.0311e-05     	|63             	|  0.00064959     	|  4.198e-05      	|
on_fit_end                         	|  2.2943e-05     	|1              	|  2.2943e-05     	|  1.4827e-06     	|
on_before_accelerator_backend_setup	|  1.9957e-05     	|1              	|  1.9957e-05     	|  1.2897e-06     	|
on_fit_start                       	|  1.657e-05      	|1              	|  1.657e-05      	|  1.0708e-06     	|
on_configure_sharded_model         	|  1.3044e-05     	|1              	|  1.3044e-05     	|  8.4297e-07     	|
setup                              	|  1.2823e-05     	|1              	|  1.2823e-05     	|  8.287e-07      	|
on_pretrain_routine_end            	|  1.2733e-05     	|1              	|  1.2733e-05     	|  8.2287e-07     	|
teardown                           	|  1.1281e-05     	|1              	|  1.1281e-05     	|  7.2903e-07     	|
prepare_data                       	|  8.3861e-06     	|1              	|  8.3861e-06     	|  5.4195e-07     	|
configure_callbacks                	|  6.201e-06      	|1              	|  6.201e-06      	|  4.0074e-07     	|
on_train_dataloader                	|  4.3879e-06     	|1              	|  4.3879e-06     	|  2.8357e-07     	|
on_val_dataloader                  	|  4.3481e-06     	|1              	|  4.3481e-06     	|  2.81e-07       	|
configure_sharded_model            	|  3.9071e-06     	|1              	|  3.9071e-06     	|  2.525e-07      	|
