VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  658.4          	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  3.6363         	|167            	|  607.26         	|  92.233         	|
run_training_batch                 	|  0.24513        	|1000           	|  245.13         	|  37.231         	|
optimizer_step_with_closure_0      	|  0.24301        	|1000           	|  243.01         	|  36.909         	|
training_step_and_backward         	|  0.18282        	|1000           	|  182.82         	|  27.768         	|
model_forward                      	|  0.095448       	|1000           	|  95.448         	|  14.497         	|
training_step                      	|  0.095274       	|1000           	|  95.274         	|  14.471         	|
backward                           	|  0.086063       	|1000           	|  86.063         	|  13.071         	|
get_train_batch                    	|  0.067695       	|1167           	|  79.0           	|  11.999         	|
fetch_next_train_batch             	|  0.067669       	|1167           	|  78.969         	|  11.994         	|
run_validate_evaluation            	|  16.479         	|1              	|  16.479         	|  2.5028         	|
evaluation_step_and_end            	|  0.1058         	|125            	|  13.225         	|  2.0086         	|
validation_step                    	|  0.10573        	|125            	|  13.216         	|  2.0073         	|
on_train_batch_end                 	|  0.0016428      	|1000           	|  1.6428         	|  0.24952        	|
zero_grad                          	|  0.0012816      	|1000           	|  1.2816         	|  0.19465        	|
get_validate_batch                 	|  0.0068755      	|126            	|  0.86632        	|  0.13158        	|
fetch_next_validate_batch          	|  0.0068581      	|126            	|  0.86412        	|  0.13124        	|
training_batch_to_device           	|  0.00035069     	|1000           	|  0.35069        	|  0.053264       	|
on_train_batch_start               	|  0.00015384     	|1000           	|  0.15384        	|  0.023366       	|
on_train_epoch_end                 	|  0.00032594     	|167            	|  0.054432       	|  0.0082672      	|
on_train_epoch_start               	|  0.00030841     	|167            	|  0.051505       	|  0.0078227      	|
on_train_end                       	|  0.046492       	|1              	|  0.046492       	|  0.0070613      	|
on_batch_start                     	|  2.8295e-05     	|1000           	|  0.028295       	|  0.0042975      	|
evaluation_batch_to_device         	|  0.00021129     	|125            	|  0.026411       	|  0.0040114      	|
on_pretrain_routine_start          	|  0.022481       	|1              	|  0.022481       	|  0.0034145      	|
on_after_backward                  	|  1.8832e-05     	|1000           	|  0.018832       	|  0.0028602      	|
on_before_zero_grad                	|  1.6323e-05     	|1000           	|  0.016323       	|  0.0024792      	|
on_batch_end                       	|  1.4934e-05     	|1000           	|  0.014934       	|  0.0022683      	|
on_before_backward                 	|  1.211e-05      	|1000           	|  0.01211        	|  0.0018392      	|
on_before_optimizer_step           	|  1.2084e-05     	|1000           	|  0.012084       	|  0.0018354      	|
configure_optimizers               	|  0.011207       	|1              	|  0.011207       	|  0.0017021      	|
training_step_end                  	|  6.6849e-06     	|1000           	|  0.0066849      	|  0.0010153      	|
on_validation_model_eval           	|  0.0048263      	|1              	|  0.0048263      	|  0.00073304     	|
on_validation_batch_end            	|  2.6924e-05     	|125            	|  0.0033655      	|  0.00051117     	|
on_epoch_end                       	|  1.143e-05      	|168            	|  0.0019202      	|  0.00029165     	|
on_train_start                     	|  0.0017713      	|1              	|  0.0017713      	|  0.00026903     	|
on_epoch_start                     	|  1.0536e-05     	|168            	|  0.00177        	|  0.00026884     	|
on_validation_batch_start          	|  1.2194e-05     	|125            	|  0.0015243      	|  0.00023151     	|
validation_step_end                	|  6.2037e-06     	|125            	|  0.00077547     	|  0.00011778     	|
on_validation_start                	|  0.00065894     	|1              	|  0.00065894     	|  0.00010008     	|
on_validation_end                  	|  0.00029684     	|1              	|  0.00029684     	|  4.5084e-05     	|
on_fit_end                         	|  0.00017586     	|1              	|  0.00017586     	|  2.6711e-05     	|
on_before_accelerator_backend_setup	|  2.8715e-05     	|2              	|  5.7429e-05     	|  8.7225e-06     	|
teardown                           	|  2.1785e-05     	|2              	|  4.3569e-05     	|  6.6174e-06     	|
on_fit_start                       	|  4.3029e-05     	|1              	|  4.3029e-05     	|  6.5354e-06     	|
on_configure_sharded_model         	|  2.0815e-05     	|2              	|  4.163e-05      	|  6.3229e-06     	|
setup                              	|  1.8754e-05     	|2              	|  3.7508e-05     	|  5.6968e-06     	|
on_validation_epoch_end            	|  2.8804e-05     	|1              	|  2.8804e-05     	|  4.3748e-06     	|
configure_callbacks                	|  1.4008e-05     	|2              	|  2.8016e-05     	|  4.2552e-06     	|
on_val_dataloader                  	|  1.38e-05       	|2              	|  2.7601e-05     	|  4.1921e-06     	|
prepare_data                       	|  9.3691e-06     	|2              	|  1.8738e-05     	|  2.846e-06      	|
configure_sharded_model            	|  6.3581e-06     	|2              	|  1.2716e-05     	|  1.9314e-06     	|
on_pretrain_routine_end            	|  1.1956e-05     	|1              	|  1.1956e-05     	|  1.816e-06      	|
on_validation_epoch_start          	|  1.1003e-05     	|1              	|  1.1003e-05     	|  1.6711e-06     	|
on_train_dataloader                	|  6.9924e-06     	|1              	|  6.9924e-06     	|  1.062e-06      	|
