FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  455.79         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.5921         	|167            	|  432.88         	|  94.973         	|
run_training_batch                 	|  0.37925        	|1000           	|  379.25         	|  83.207         	|
optimizer_step_with_closure_0      	|  0.37656        	|1000           	|  376.56         	|  82.615         	|
training_step_and_backward         	|  0.29712        	|1000           	|  297.12         	|  65.187         	|
model_forward                      	|  0.1721         	|1000           	|  172.1          	|  37.758         	|
training_step                      	|  0.17195        	|1000           	|  171.95         	|  37.726         	|
backward                           	|  0.1237         	|1000           	|  123.7          	|  27.14          	|
get_train_batch                    	|  0.014151       	|1167           	|  16.514         	|  3.6232         	|
fetch_next_train_batch             	|  0.014127       	|1167           	|  16.486         	|  3.6171         	|
zero_grad                          	|  0.0012904      	|1000           	|  1.2904         	|  0.2831         	|
on_train_batch_end                 	|  0.0012333      	|1000           	|  1.2333         	|  0.27057        	|
training_batch_to_device           	|  0.00031102     	|1000           	|  0.31102        	|  0.068237       	|
on_train_batch_start               	|  9.6086e-05     	|1000           	|  0.096086       	|  0.021081       	|
on_train_epoch_start               	|  0.00027686     	|167            	|  0.046235       	|  0.010144       	|
on_train_epoch_end                 	|  0.00026861     	|167            	|  0.044858       	|  0.0098416      	|
on_train_end                       	|  0.024833       	|1              	|  0.024833       	|  0.0054483      	|
on_pretrain_routine_start          	|  0.022241       	|1              	|  0.022241       	|  0.0048796      	|
on_batch_start                     	|  2.1075e-05     	|1000           	|  0.021075       	|  0.0046239      	|
on_before_zero_grad                	|  1.4966e-05     	|1000           	|  0.014966       	|  0.0032835      	|
on_batch_end                       	|  1.4189e-05     	|1000           	|  0.014189       	|  0.003113       	|
on_after_backward                  	|  1.1191e-05     	|1000           	|  0.011191       	|  0.0024552      	|
on_before_optimizer_step           	|  1.077e-05      	|1000           	|  0.01077        	|  0.0023629      	|
on_before_backward                 	|  1.0595e-05     	|1000           	|  0.010595       	|  0.0023245      	|
configure_optimizers               	|  0.0080945      	|1              	|  0.0080945      	|  0.0017759      	|
training_step_end                  	|  6.7079e-06     	|1000           	|  0.0067079      	|  0.0014717      	|
on_epoch_end                       	|  1.1019e-05     	|167            	|  0.0018401      	|  0.00040371     	|
on_epoch_start                     	|  1.0679e-05     	|167            	|  0.0017834      	|  0.00039127     	|
on_train_start                     	|  0.00072556     	|1              	|  0.00072556     	|  0.00015919     	|
on_fit_end                         	|  2.7101e-05     	|1              	|  2.7101e-05     	|  5.9459e-06     	|
on_before_accelerator_backend_setup	|  1.9587e-05     	|1              	|  1.9587e-05     	|  4.2973e-06     	|
on_fit_start                       	|  1.628e-05      	|1              	|  1.628e-05      	|  3.5718e-06     	|
setup                              	|  1.574e-05      	|1              	|  1.574e-05      	|  3.4533e-06     	|
on_configure_sharded_model         	|  1.3405e-05     	|1              	|  1.3405e-05     	|  2.941e-06      	|
teardown                           	|  1.0149e-05     	|1              	|  1.0149e-05     	|  2.2267e-06     	|
on_pretrain_routine_end            	|  9.899e-06      	|1              	|  9.899e-06      	|  2.1718e-06     	|
prepare_data                       	|  8.235e-06      	|1              	|  8.235e-06      	|  1.8067e-06     	|
configure_callbacks                	|  6.5119e-06     	|1              	|  6.5119e-06     	|  1.4287e-06     	|
configure_sharded_model            	|  4.388e-06      	|1              	|  4.388e-06      	|  9.6272e-07     	|
on_train_dataloader                	|  4.3791e-06     	|1              	|  4.3791e-06     	|  9.6076e-07     	|
on_val_dataloader                  	|  4.348e-06      	|1              	|  4.348e-06      	|  9.5394e-07     	|
FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1573.4         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.2822         	|167            	|  1550.1         	|  98.524         	|
run_training_batch                 	|  0.18791        	|8000           	|  1503.3         	|  95.546         	|
training_step_and_backward         	|  0.17772        	|8000           	|  1421.8         	|  90.366         	|
model_forward                      	|  0.094105       	|8000           	|  752.84         	|  47.849         	|
training_step                      	|  0.09397        	|8000           	|  751.76         	|  47.781         	|
backward                           	|  0.083432       	|8000           	|  667.45         	|  42.422         	|
optimizer_step_with_closure_0      	|  0.23625        	|1000           	|  236.25         	|  15.016         	|
get_train_batch                    	|  0.0033302      	|8167           	|  27.198         	|  1.7287         	|
fetch_next_train_batch             	|  0.0033127      	|8167           	|  27.055         	|  1.7195         	|
on_train_batch_end                 	|  0.001082       	|8000           	|  8.6559         	|  0.55015        	|
training_batch_to_device           	|  0.00021816     	|8000           	|  1.7453         	|  0.11093        	|
zero_grad                          	|  0.0013352      	|1000           	|  1.3352         	|  0.084863       	|
on_train_batch_start               	|  8.9803e-05     	|8000           	|  0.71842        	|  0.045662       	|
on_batch_start                     	|  1.5038e-05     	|8000           	|  0.1203         	|  0.0076464      	|
on_batch_end                       	|  1.3342e-05     	|8000           	|  0.10674        	|  0.0067841      	|
on_before_backward                 	|  1.266e-05      	|8000           	|  0.10128        	|  0.0064374      	|
on_after_backward                  	|  1.1234e-05     	|8000           	|  0.08987        	|  0.005712       	|
on_train_epoch_start               	|  0.00029176     	|167            	|  0.048724       	|  0.0030968      	|
training_step_end                  	|  5.887e-06      	|8000           	|  0.047096       	|  0.0029933      	|
on_train_epoch_end                 	|  0.00026155     	|167            	|  0.043678       	|  0.0027761      	|
on_train_end                       	|  0.025969       	|1              	|  0.025969       	|  0.0016506      	|
on_pretrain_routine_start          	|  0.022674       	|1              	|  0.022674       	|  0.0014411      	|
on_before_zero_grad                	|  1.5428e-05     	|1000           	|  0.015428       	|  0.00098059     	|
on_before_optimizer_step           	|  1.268e-05      	|1000           	|  0.01268        	|  0.00080592     	|
configure_optimizers               	|  0.0088318      	|1              	|  0.0088318      	|  0.00056133     	|
on_epoch_start                     	|  1.0696e-05     	|167            	|  0.0017862      	|  0.00011353     	|
on_epoch_end                       	|  1.0427e-05     	|167            	|  0.0017413      	|  0.00011068     	|
on_train_start                     	|  0.00046952     	|1              	|  0.00046952     	|  2.9842e-05     	|
on_fit_end                         	|  2.125e-05      	|1              	|  2.125e-05      	|  1.3506e-06     	|
on_before_accelerator_backend_setup	|  2.0609e-05     	|1              	|  2.0609e-05     	|  1.3099e-06     	|
on_fit_start                       	|  1.7843e-05     	|1              	|  1.7843e-05     	|  1.1341e-06     	|
on_configure_sharded_model         	|  1.4267e-05     	|1              	|  1.4267e-05     	|  9.0678e-07     	|
setup                              	|  1.3565e-05     	|1              	|  1.3565e-05     	|  8.6217e-07     	|
on_pretrain_routine_end            	|  1.3054e-05     	|1              	|  1.3054e-05     	|  8.2968e-07     	|
teardown                           	|  1.0129e-05     	|1              	|  1.0129e-05     	|  6.4379e-07     	|
prepare_data                       	|  8.906e-06      	|1              	|  8.906e-06      	|  5.6605e-07     	|
configure_callbacks                	|  6.5919e-06     	|1              	|  6.5919e-06     	|  4.1897e-07     	|
on_val_dataloader                  	|  5.1791e-06     	|1              	|  5.1791e-06     	|  3.2917e-07     	|
configure_sharded_model            	|  4.749e-06      	|1              	|  4.749e-06      	|  3.0184e-07     	|
on_train_dataloader                	|  4.549e-06      	|1              	|  4.549e-06      	|  2.8913e-07     	|
