FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  452.25         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.5716         	|167            	|  429.45         	|  94.958         	|
run_training_batch                 	|  0.37587        	|1000           	|  375.87         	|  83.111         	|
optimizer_step_with_closure_0      	|  0.37313        	|1000           	|  373.13         	|  82.504         	|
training_step_and_backward         	|  0.29413        	|1000           	|  294.13         	|  65.036         	|
model_forward                      	|  0.17059        	|1000           	|  170.59         	|  37.72          	|
training_step                      	|  0.17044        	|1000           	|  170.44         	|  37.687         	|
backward                           	|  0.12219        	|1000           	|  122.19         	|  27.018         	|
get_train_batch                    	|  0.014131       	|1167           	|  16.491         	|  3.6464         	|
fetch_next_train_batch             	|  0.014107       	|1167           	|  16.463         	|  3.6403         	|
zero_grad                          	|  0.0013221      	|1000           	|  1.3221         	|  0.29234        	|
on_train_batch_end                 	|  0.0012363      	|1000           	|  1.2363         	|  0.27336        	|
training_batch_to_device           	|  0.00031446     	|1000           	|  0.31446        	|  0.069532       	|
on_train_batch_start               	|  9.513e-05      	|1000           	|  0.09513        	|  0.021035       	|
on_train_epoch_start               	|  0.00027926     	|167            	|  0.046636       	|  0.010312       	|
on_train_epoch_end                 	|  0.00026533     	|167            	|  0.04431        	|  0.0097976      	|
on_train_end                       	|  0.024651       	|1              	|  0.024651       	|  0.0054507      	|
on_pretrain_routine_start          	|  0.022596       	|1              	|  0.022596       	|  0.0049963      	|
on_batch_start                     	|  2.1059e-05     	|1000           	|  0.021059       	|  0.0046564      	|
on_before_zero_grad                	|  1.5399e-05     	|1000           	|  0.015399       	|  0.0034049      	|
on_batch_end                       	|  1.3804e-05     	|1000           	|  0.013804       	|  0.0030522      	|
on_after_backward                  	|  1.1807e-05     	|1000           	|  0.011807       	|  0.0026107      	|
on_before_optimizer_step           	|  1.1354e-05     	|1000           	|  0.011354       	|  0.0025105      	|
on_before_backward                 	|  1.072e-05      	|1000           	|  0.01072        	|  0.0023703      	|
configure_optimizers               	|  0.0085719      	|1              	|  0.0085719      	|  0.0018954      	|
training_step_end                  	|  6.9868e-06     	|1000           	|  0.0069868      	|  0.0015449      	|
on_epoch_end                       	|  1.0528e-05     	|167            	|  0.0017582      	|  0.00038876     	|
on_epoch_start                     	|  1.0396e-05     	|167            	|  0.0017361      	|  0.00038388     	|
on_train_start                     	|  0.00075695     	|1              	|  0.00075695     	|  0.00016737     	|
on_before_accelerator_backend_setup	|  2.4255e-05     	|1              	|  2.4255e-05     	|  5.3631e-06     	|
on_fit_end                         	|  2.3484e-05     	|1              	|  2.3484e-05     	|  5.1926e-06     	|
on_fit_start                       	|  1.6651e-05     	|1              	|  1.6651e-05     	|  3.6818e-06     	|
setup                              	|  1.5819e-05     	|1              	|  1.5819e-05     	|  3.4978e-06     	|
on_configure_sharded_model         	|  1.4557e-05     	|1              	|  1.4557e-05     	|  3.2188e-06     	|
on_pretrain_routine_end            	|  1.082e-05      	|1              	|  1.082e-05      	|  2.3925e-06     	|
teardown                           	|  1.0088e-05     	|1              	|  1.0088e-05     	|  2.2306e-06     	|
prepare_data                       	|  9.648e-06      	|1              	|  9.648e-06      	|  2.1333e-06     	|
configure_callbacks                	|  5.691e-06      	|1              	|  5.691e-06      	|  1.2584e-06     	|
configure_sharded_model            	|  4.779e-06      	|1              	|  4.779e-06      	|  1.0567e-06     	|
on_val_dataloader                  	|  4.6589e-06     	|1              	|  4.6589e-06     	|  1.0302e-06     	|
on_train_dataloader                	|  4.569e-06      	|1              	|  4.569e-06      	|  1.0103e-06     	|
FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1557.1         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.1836         	|167            	|  1533.7         	|  98.496         	|
run_training_batch                 	|  0.18581        	|8000           	|  1486.5         	|  95.465         	|
training_step_and_backward         	|  0.17573        	|8000           	|  1405.8         	|  90.286         	|
model_forward                      	|  0.094059       	|8000           	|  752.47         	|  48.326         	|
training_step                      	|  0.093926       	|8000           	|  751.41         	|  48.257         	|
backward                           	|  0.081489       	|8000           	|  651.91         	|  41.868         	|
optimizer_step_with_closure_0      	|  0.23367        	|1000           	|  233.67         	|  15.007         	|
get_train_batch                    	|  0.0033553      	|8167           	|  27.403         	|  1.7599         	|
fetch_next_train_batch             	|  0.0033381      	|8167           	|  27.263         	|  1.7509         	|
on_train_batch_end                 	|  0.0010768      	|8000           	|  8.6143         	|  0.55323        	|
training_batch_to_device           	|  0.00021962     	|8000           	|  1.757          	|  0.11284        	|
zero_grad                          	|  0.0013094      	|1000           	|  1.3094         	|  0.084091       	|
on_train_batch_start               	|  9.0942e-05     	|8000           	|  0.72753        	|  0.046724       	|
on_batch_start                     	|  1.5402e-05     	|8000           	|  0.12321        	|  0.007913       	|
on_batch_end                       	|  1.2865e-05     	|8000           	|  0.10292        	|  0.0066097      	|
on_before_backward                 	|  1.2739e-05     	|8000           	|  0.10192        	|  0.0065453      	|
on_after_backward                  	|  1.1335e-05     	|8000           	|  0.090676       	|  0.0058234      	|
on_train_epoch_start               	|  0.00029052     	|167            	|  0.048517       	|  0.0031159      	|
training_step_end                  	|  5.7425e-06     	|8000           	|  0.04594        	|  0.0029504      	|
on_train_epoch_end                 	|  0.00025643     	|167            	|  0.042824       	|  0.0027503      	|
on_train_end                       	|  0.026792       	|1              	|  0.026792       	|  0.0017206      	|
on_pretrain_routine_start          	|  0.022945       	|1              	|  0.022945       	|  0.0014736      	|
on_before_zero_grad                	|  1.5749e-05     	|1000           	|  0.015749       	|  0.0010114      	|
on_before_optimizer_step           	|  1.2436e-05     	|1000           	|  0.012436       	|  0.00079864     	|
configure_optimizers               	|  0.0084384      	|1              	|  0.0084384      	|  0.00054193     	|
on_epoch_end                       	|  1.1097e-05     	|167            	|  0.0018531      	|  0.00011901     	|
on_epoch_start                     	|  1.0555e-05     	|167            	|  0.0017627      	|  0.00011321     	|
on_train_start                     	|  0.00075286     	|1              	|  0.00075286     	|  4.835e-05      	|
on_fit_end                         	|  2.5157e-05     	|1              	|  2.5157e-05     	|  1.6156e-06     	|
on_before_accelerator_backend_setup	|  2.2282e-05     	|1              	|  2.2282e-05     	|  1.431e-06      	|
on_fit_start                       	|  1.7833e-05     	|1              	|  1.7833e-05     	|  1.1453e-06     	|
on_configure_sharded_model         	|  1.4036e-05     	|1              	|  1.4036e-05     	|  9.0143e-07     	|
setup                              	|  1.3665e-05     	|1              	|  1.3665e-05     	|  8.7761e-07     	|
teardown                           	|  1.1862e-05     	|1              	|  1.1862e-05     	|  7.6179e-07     	|
on_pretrain_routine_end            	|  1.1792e-05     	|1              	|  1.1792e-05     	|  7.5731e-07     	|
prepare_data                       	|  7.404e-06      	|1              	|  7.404e-06      	|  4.755e-07      	|
configure_callbacks                	|  6.312e-06      	|1              	|  6.312e-06      	|  4.0538e-07     	|
configure_sharded_model            	|  4.709e-06      	|1              	|  4.709e-06      	|  3.0242e-07     	|
on_val_dataloader                  	|  4.2981e-06     	|1              	|  4.2981e-06     	|  2.7603e-07     	|
on_train_dataloader                	|  4.238e-06      	|1              	|  4.238e-06      	|  2.7217e-07     	|
