FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  450.24         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.5595         	|167            	|  427.44         	|  94.937         	|
run_training_batch                 	|  0.37403        	|1000           	|  374.03         	|  83.073         	|
optimizer_step_with_closure_0      	|  0.37135        	|1000           	|  371.35         	|  82.479         	|
training_step_and_backward         	|  0.29274        	|1000           	|  292.74         	|  65.019         	|
model_forward                      	|  0.16994        	|1000           	|  169.94         	|  37.744         	|
training_step                      	|  0.16979        	|1000           	|  169.79         	|  37.711         	|
backward                           	|  0.12146        	|1000           	|  121.46         	|  26.977         	|
get_train_batch                    	|  0.013997       	|1167           	|  16.334         	|  3.6279         	|
fetch_next_train_batch             	|  0.013974       	|1167           	|  16.307         	|  3.6219         	|
zero_grad                          	|  0.0013145      	|1000           	|  1.3145         	|  0.29196        	|
on_train_batch_end                 	|  0.0012413      	|1000           	|  1.2413         	|  0.27569        	|
training_batch_to_device           	|  0.00031829     	|1000           	|  0.31829        	|  0.070694       	|
on_train_batch_start               	|  9.6186e-05     	|1000           	|  0.096186       	|  0.021363       	|
on_train_epoch_start               	|  0.00027518     	|167            	|  0.045955       	|  0.010207       	|
on_train_epoch_end                 	|  0.00026083     	|167            	|  0.043559       	|  0.0096747      	|
on_train_end                       	|  0.025576       	|1              	|  0.025576       	|  0.0056805      	|
on_pretrain_routine_start          	|  0.022584       	|1              	|  0.022584       	|  0.005016       	|
on_batch_start                     	|  2.1509e-05     	|1000           	|  0.021509       	|  0.0047773      	|
on_before_zero_grad                	|  1.529e-05      	|1000           	|  0.01529        	|  0.0033959      	|
on_batch_end                       	|  1.4437e-05     	|1000           	|  0.014437       	|  0.0032066      	|
on_after_backward                  	|  1.1484e-05     	|1000           	|  0.011484       	|  0.0025506      	|
on_before_backward                 	|  1.1355e-05     	|1000           	|  0.011355       	|  0.002522       	|
on_before_optimizer_step           	|  1.1354e-05     	|1000           	|  0.011354       	|  0.0025219      	|
configure_optimizers               	|  0.0083916      	|1              	|  0.0083916      	|  0.0018638      	|
training_step_end                  	|  7.2327e-06     	|1000           	|  0.0072327      	|  0.0016064      	|
on_epoch_start                     	|  1.0609e-05     	|167            	|  0.0017717      	|  0.00039351     	|
on_epoch_end                       	|  1.0434e-05     	|167            	|  0.0017425      	|  0.00038702     	|
on_train_start                     	|  0.0007298      	|1              	|  0.0007298      	|  0.00016209     	|
on_fit_end                         	|  2.1781e-05     	|1              	|  2.1781e-05     	|  4.8377e-06     	|
on_before_accelerator_backend_setup	|  1.9286e-05     	|1              	|  1.9286e-05     	|  4.2835e-06     	|
on_fit_start                       	|  1.7122e-05     	|1              	|  1.7122e-05     	|  3.8029e-06     	|
setup                              	|  1.5248e-05     	|1              	|  1.5248e-05     	|  3.3866e-06     	|
on_configure_sharded_model         	|  1.3365e-05     	|1              	|  1.3365e-05     	|  2.9684e-06     	|
teardown                           	|  1.1341e-05     	|1              	|  1.1341e-05     	|  2.5189e-06     	|
on_pretrain_routine_end            	|  9.709e-06      	|1              	|  9.709e-06      	|  2.1564e-06     	|
prepare_data                       	|  7.2541e-06     	|1              	|  7.2541e-06     	|  1.6112e-06     	|
configure_callbacks                	|  5.981e-06      	|1              	|  5.981e-06      	|  1.3284e-06     	|
on_val_dataloader                  	|  4.518e-06      	|1              	|  4.518e-06      	|  1.0035e-06     	|
configure_sharded_model            	|  4.1281e-06     	|1              	|  4.1281e-06     	|  9.1687e-07     	|
on_train_dataloader                	|  3.887e-06      	|1              	|  3.887e-06      	|  8.6332e-07     	|
FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1550.8         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.145          	|167            	|  1527.2         	|  98.482         	|
run_training_batch                 	|  0.185          	|8000           	|  1480.0         	|  95.439         	|
training_step_and_backward         	|  0.175          	|8000           	|  1400.0         	|  90.279         	|
model_forward                      	|  0.093997       	|8000           	|  751.98         	|  48.491         	|
training_step                      	|  0.093863       	|8000           	|  750.9          	|  48.422         	|
backward                           	|  0.08082        	|8000           	|  646.56         	|  41.693         	|
optimizer_step_with_closure_0      	|  0.23253        	|1000           	|  232.53         	|  14.995         	|
get_train_batch                    	|  0.0033566      	|8167           	|  27.413         	|  1.7677         	|
fetch_next_train_batch             	|  0.0033387      	|8167           	|  27.267         	|  1.7583         	|
on_train_batch_end                 	|  0.0010758      	|8000           	|  8.6063         	|  0.55497        	|
training_batch_to_device           	|  0.00021793     	|8000           	|  1.7435         	|  0.11243        	|
zero_grad                          	|  0.0013255      	|1000           	|  1.3255         	|  0.085472       	|
on_train_batch_start               	|  9.0501e-05     	|8000           	|  0.72401        	|  0.046687       	|
on_batch_start                     	|  1.4879e-05     	|8000           	|  0.11903        	|  0.0076759      	|
on_batch_end                       	|  1.334e-05      	|8000           	|  0.10672        	|  0.0068816      	|
on_before_backward                 	|  1.304e-05      	|8000           	|  0.10432        	|  0.0067273      	|
on_after_backward                  	|  1.1363e-05     	|8000           	|  0.090901       	|  0.0058617      	|
on_train_epoch_start               	|  0.0002921      	|167            	|  0.048781       	|  0.0031456      	|
training_step_end                  	|  5.9931e-06     	|8000           	|  0.047945       	|  0.0030917      	|
on_train_epoch_end                 	|  0.00026057     	|167            	|  0.043515       	|  0.002806       	|
on_train_end                       	|  0.025303       	|1              	|  0.025303       	|  0.0016317      	|
on_pretrain_routine_start          	|  0.022868       	|1              	|  0.022868       	|  0.0014746      	|
on_before_zero_grad                	|  1.6519e-05     	|1000           	|  0.016519       	|  0.0010653      	|
on_before_optimizer_step           	|  1.4494e-05     	|1000           	|  0.014494       	|  0.00093465     	|
configure_optimizers               	|  0.0085078      	|1              	|  0.0085078      	|  0.00054862     	|
on_epoch_start                     	|  1.1248e-05     	|167            	|  0.0018784      	|  0.00012113     	|
on_epoch_end                       	|  1.1055e-05     	|167            	|  0.0018461      	|  0.00011905     	|
on_train_start                     	|  0.00076435     	|1              	|  0.00076435     	|  4.9289e-05     	|
on_fit_end                         	|  2.2753e-05     	|1              	|  2.2753e-05     	|  1.4672e-06     	|
on_before_accelerator_backend_setup	|  2.0287e-05     	|1              	|  2.0287e-05     	|  1.3082e-06     	|
on_fit_start                       	|  1.7933e-05     	|1              	|  1.7933e-05     	|  1.1564e-06     	|
setup                              	|  1.4176e-05     	|1              	|  1.4176e-05     	|  9.1413e-07     	|
on_configure_sharded_model         	|  1.3645e-05     	|1              	|  1.3645e-05     	|  8.799e-07      	|
on_pretrain_routine_end            	|  1.1782e-05     	|1              	|  1.1782e-05     	|  7.5975e-07     	|
teardown                           	|  1.0429e-05     	|1              	|  1.0429e-05     	|  6.7252e-07     	|
prepare_data                       	|  6.943e-06      	|1              	|  6.943e-06      	|  4.4772e-07     	|
configure_callbacks                	|  6.6222e-06     	|1              	|  6.6222e-06     	|  4.2703e-07     	|
on_val_dataloader                  	|  4.7181e-06     	|1              	|  4.7181e-06     	|  3.0424e-07     	|
configure_sharded_model            	|  4.5679e-06     	|1              	|  4.5679e-06     	|  2.9456e-07     	|
on_train_dataloader                	|  4.268e-06      	|1              	|  4.268e-06      	|  2.7522e-07     	|
