FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  382.72         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.6973         	|63             	|  358.93         	|  93.785         	|
run_training_batch                 	|  0.33231        	|1000           	|  332.31         	|  86.83          	|
optimizer_step_with_closure_0      	|  0.32966        	|1000           	|  329.66         	|  86.137         	|
training_step_and_backward         	|  0.24537        	|1000           	|  245.37         	|  64.113         	|
model_forward                      	|  0.15246        	|1000           	|  152.46         	|  39.836         	|
training_step                      	|  0.15232        	|1000           	|  152.32         	|  39.8           	|
backward                           	|  0.091552       	|1000           	|  91.552         	|  23.922         	|
get_train_batch                    	|  0.0080566      	|1063           	|  8.5641         	|  2.2377         	|
fetch_next_train_batch             	|  0.008038       	|1063           	|  8.5444         	|  2.2326         	|
zero_grad                          	|  0.0013333      	|1000           	|  1.3333         	|  0.34839        	|
on_train_batch_end                 	|  0.001204       	|1000           	|  1.204          	|  0.31459        	|
training_batch_to_device           	|  0.00023198     	|1000           	|  0.23198        	|  0.060614       	|
on_train_batch_start               	|  9.2995e-05     	|1000           	|  0.092995       	|  0.024299       	|
on_train_end                       	|  0.025482       	|1              	|  0.025482       	|  0.0066581      	|
on_pretrain_routine_start          	|  0.023743       	|1              	|  0.023743       	|  0.0062037      	|
on_train_epoch_start               	|  0.00029386     	|63             	|  0.018513       	|  0.0048373      	|
on_batch_start                     	|  1.8103e-05     	|1000           	|  0.018103       	|  0.0047302      	|
on_train_epoch_end                 	|  0.00028157     	|63             	|  0.017739       	|  0.004635       	|
on_before_zero_grad                	|  1.4654e-05     	|1000           	|  0.014654       	|  0.0038289      	|
on_batch_end                       	|  1.3352e-05     	|1000           	|  0.013352       	|  0.0034887      	|
on_after_backward                  	|  1.1704e-05     	|1000           	|  0.011704       	|  0.0030582      	|
on_before_optimizer_step           	|  1.0756e-05     	|1000           	|  0.010756       	|  0.0028103      	|
on_before_backward                 	|  1.071e-05      	|1000           	|  0.01071        	|  0.0027985      	|
configure_optimizers               	|  0.0080753      	|1              	|  0.0080753      	|  0.00211        	|
training_step_end                  	|  6.2007e-06     	|1000           	|  0.0062007      	|  0.0016202      	|
on_train_start                     	|  0.00072989     	|1              	|  0.00072989     	|  0.00019071     	|
on_epoch_start                     	|  1.116e-05      	|63             	|  0.00070307     	|  0.0001837      	|
on_epoch_end                       	|  1.0581e-05     	|63             	|  0.00066663     	|  0.00017418     	|
on_fit_end                         	|  2.3525e-05     	|1              	|  2.3525e-05     	|  6.1469e-06     	|
on_before_accelerator_backend_setup	|  1.9236e-05     	|1              	|  1.9236e-05     	|  5.0262e-06     	|
on_fit_start                       	|  1.5849e-05     	|1              	|  1.5849e-05     	|  4.1412e-06     	|
on_configure_sharded_model         	|  1.3445e-05     	|1              	|  1.3445e-05     	|  3.5131e-06     	|
setup                              	|  1.3385e-05     	|1              	|  1.3385e-05     	|  3.4974e-06     	|
teardown                           	|  9.96e-06       	|1              	|  9.96e-06       	|  2.6025e-06     	|
on_pretrain_routine_end            	|  9.6781e-06     	|1              	|  9.6781e-06     	|  2.5288e-06     	|
prepare_data                       	|  7.894e-06      	|1              	|  7.894e-06      	|  2.0626e-06     	|
configure_callbacks                	|  5.029e-06      	|1              	|  5.029e-06      	|  1.314e-06      	|
on_val_dataloader                  	|  4.749e-06      	|1              	|  4.749e-06      	|  1.2409e-06     	|
on_train_dataloader                	|  3.968e-06      	|1              	|  3.968e-06      	|  1.0368e-06     	|
configure_sharded_model            	|  3.9571e-06     	|1              	|  3.9571e-06     	|  1.0339e-06     	|
FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1547.9         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.204         	|63             	|  1524.9         	|  98.511         	|
run_training_batch                 	|  0.1862         	|8000           	|  1489.6         	|  96.235         	|
training_step_and_backward         	|  0.17607        	|8000           	|  1408.6         	|  91.0           	|
model_forward                      	|  0.092309       	|8000           	|  738.47         	|  47.708         	|
training_step                      	|  0.092176       	|8000           	|  737.41         	|  47.639         	|
backward                           	|  0.083581       	|8000           	|  668.65         	|  43.197         	|
optimizer_step_with_closure_0      	|  0.23525        	|1000           	|  235.25         	|  15.198         	|
get_train_batch                    	|  0.002295       	|8063           	|  18.505         	|  1.1955         	|
fetch_next_train_batch             	|  0.0022783      	|8063           	|  18.37          	|  1.1868         	|
on_train_batch_end                 	|  0.0010732      	|8000           	|  8.5852         	|  0.55464        	|
training_batch_to_device           	|  0.00021378     	|8000           	|  1.7103         	|  0.11049        	|
zero_grad                          	|  0.0013245      	|1000           	|  1.3245         	|  0.085567       	|
on_train_batch_start               	|  9.053e-05      	|8000           	|  0.72424        	|  0.046788       	|
on_batch_start                     	|  1.5392e-05     	|8000           	|  0.12314        	|  0.007955       	|
on_batch_end                       	|  1.2648e-05     	|8000           	|  0.10119        	|  0.006537       	|
on_before_backward                 	|  1.2415e-05     	|8000           	|  0.099318       	|  0.0064163      	|
on_after_backward                  	|  1.1346e-05     	|8000           	|  0.090765       	|  0.0058637      	|
training_step_end                  	|  5.8202e-06     	|8000           	|  0.046561       	|  0.003008       	|
on_train_end                       	|  0.025578       	|1              	|  0.025578       	|  0.0016524      	|
on_pretrain_routine_start          	|  0.022251       	|1              	|  0.022251       	|  0.0014375      	|
on_train_epoch_start               	|  0.00030256     	|63             	|  0.019062       	|  0.0012314      	|
on_train_epoch_end                 	|  0.00028553     	|63             	|  0.017988       	|  0.0011621      	|
on_before_zero_grad                	|  1.4697e-05     	|1000           	|  0.014697       	|  0.00094948     	|
on_before_optimizer_step           	|  1.1326e-05     	|1000           	|  0.011326       	|  0.00073167     	|
configure_optimizers               	|  0.0085258      	|1              	|  0.0085258      	|  0.0005508      	|
on_train_start                     	|  0.00071599     	|1              	|  0.00071599     	|  4.6255e-05     	|
on_epoch_end                       	|  1.1062e-05     	|63             	|  0.0006969      	|  4.5022e-05     	|
on_epoch_start                     	|  1.0455e-05     	|63             	|  0.00065864     	|  4.2551e-05     	|
on_before_accelerator_backend_setup	|  2.2413e-05     	|1              	|  2.2413e-05     	|  1.448e-06      	|
on_fit_end                         	|  2.2342e-05     	|1              	|  2.2342e-05     	|  1.4434e-06     	|
on_fit_start                       	|  1.7273e-05     	|1              	|  1.7273e-05     	|  1.1159e-06     	|
on_configure_sharded_model         	|  1.3685e-05     	|1              	|  1.3685e-05     	|  8.841e-07      	|
setup                              	|  1.3104e-05     	|1              	|  1.3104e-05     	|  8.4657e-07     	|
on_pretrain_routine_end            	|  1.2563e-05     	|1              	|  1.2563e-05     	|  8.116e-07      	|
teardown                           	|  1.0058e-05     	|1              	|  1.0058e-05     	|  6.4978e-07     	|
prepare_data                       	|  9.167e-06      	|1              	|  9.167e-06      	|  5.9222e-07     	|
configure_callbacks                	|  6.3421e-06     	|1              	|  6.3421e-06     	|  4.0972e-07     	|
configure_sharded_model            	|  4.749e-06      	|1              	|  4.749e-06      	|  3.068e-07      	|
on_val_dataloader                  	|  4.7488e-06     	|1              	|  4.7488e-06     	|  3.0679e-07     	|
on_train_dataloader                	|  4.258e-06      	|1              	|  4.258e-06      	|  2.7508e-07     	|
