VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  594.6          	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  8.2735         	|63             	|  521.23         	|  87.661         	|
run_training_batch                 	|  0.38312        	|1000           	|  383.12         	|  64.434         	|
optimizer_step_with_closure_0      	|  0.37922        	|1000           	|  379.22         	|  63.778         	|
training_step_and_backward         	|  0.27327        	|1000           	|  273.27         	|  45.96          	|
model_forward                      	|  0.13772        	|1000           	|  137.72         	|  23.162         	|
training_step                      	|  0.13754        	|1000           	|  137.54         	|  23.132         	|
backward                           	|  0.13403        	|1000           	|  134.03         	|  22.542         	|
get_train_batch                    	|  0.029969       	|1063           	|  31.857         	|  5.3578         	|
fetch_next_train_batch             	|  0.029945       	|1063           	|  31.832         	|  5.3535         	|
run_validate_evaluation            	|  31.109         	|1              	|  31.109         	|  5.232          	|
evaluation_step_and_end            	|  0.22324        	|125            	|  27.905         	|  4.6931         	|
validation_step                    	|  0.22314        	|125            	|  27.893         	|  4.691          	|
on_train_batch_end                 	|  0.0020565      	|1000           	|  2.0565         	|  0.34587        	|
zero_grad                          	|  0.0014887      	|1000           	|  1.4887         	|  0.25038        	|
get_validate_batch                 	|  0.0063853      	|126            	|  0.80455        	|  0.13531        	|
fetch_next_validate_batch          	|  0.0063594      	|126            	|  0.80128        	|  0.13476        	|
training_batch_to_device           	|  0.00027058     	|1000           	|  0.27058        	|  0.045507       	|
on_train_batch_start               	|  9.5953e-05     	|1000           	|  0.095953       	|  0.016137       	|
on_train_end                       	|  0.049991       	|1              	|  0.049991       	|  0.0084075      	|
evaluation_batch_to_device         	|  0.00030496     	|125            	|  0.03812        	|  0.0064111      	|
on_train_epoch_end                 	|  0.00035566     	|63             	|  0.022407       	|  0.0037684      	|
on_train_epoch_start               	|  0.00035558     	|63             	|  0.022401       	|  0.0037675      	|
on_pretrain_routine_start          	|  0.021981       	|1              	|  0.021981       	|  0.0036968      	|
on_batch_start                     	|  2.168e-05      	|1000           	|  0.02168        	|  0.0036462      	|
on_after_backward                  	|  2.1001e-05     	|1000           	|  0.021001       	|  0.003532       	|
on_before_zero_grad                	|  1.5872e-05     	|1000           	|  0.015872       	|  0.0026693      	|
on_batch_end                       	|  1.584e-05      	|1000           	|  0.01584        	|  0.002664       	|
on_before_optimizer_step           	|  1.3552e-05     	|1000           	|  0.013552       	|  0.0022792      	|
on_before_backward                 	|  1.2445e-05     	|1000           	|  0.012445       	|  0.0020929      	|
configure_optimizers               	|  0.010555       	|1              	|  0.010555       	|  0.0017752      	|
training_step_end                  	|  7.1887e-06     	|1000           	|  0.0071887      	|  0.001209       	|
on_validation_batch_end            	|  3.7739e-05     	|125            	|  0.0047174      	|  0.00079338     	|
on_validation_model_eval           	|  0.0042297      	|1              	|  0.0042297      	|  0.00071135     	|
on_validation_batch_start          	|  1.5454e-05     	|125            	|  0.0019318      	|  0.00032489     	|
on_train_start                     	|  0.0018664      	|1              	|  0.0018664      	|  0.00031389     	|
validation_step_end                	|  8.9064e-06     	|125            	|  0.0011133      	|  0.00018724     	|
on_epoch_end                       	|  1.2817e-05     	|64             	|  0.0008203      	|  0.00013796     	|
on_epoch_start                     	|  1.1439e-05     	|64             	|  0.00073209     	|  0.00012312     	|
on_validation_start                	|  0.00054401     	|1              	|  0.00054401     	|  9.1492e-05     	|
on_validation_end                  	|  0.00018889     	|1              	|  0.00018889     	|  3.1768e-05     	|
teardown                           	|  3.6235e-05     	|2              	|  7.247e-05      	|  1.2188e-05     	|
on_before_accelerator_backend_setup	|  2.2806e-05     	|2              	|  4.5612e-05     	|  7.6712e-06     	|
on_fit_start                       	|  4.5383e-05     	|1              	|  4.5383e-05     	|  7.6326e-06     	|
on_configure_sharded_model         	|  2.2288e-05     	|2              	|  4.4577e-05     	|  7.497e-06      	|
setup                              	|  2.0211e-05     	|2              	|  4.0421e-05     	|  6.7981e-06     	|
on_validation_epoch_end            	|  3.5729e-05     	|1              	|  3.5729e-05     	|  6.009e-06      	|
on_fit_end                         	|  2.5088e-05     	|1              	|  2.5088e-05     	|  4.2193e-06     	|
on_val_dataloader                  	|  1.1615e-05     	|2              	|  2.3231e-05     	|  3.907e-06      	|
on_validation_epoch_start          	|  1.5164e-05     	|1              	|  1.5164e-05     	|  2.5503e-06     	|
configure_sharded_model            	|  7.2233e-06     	|2              	|  1.4447e-05     	|  2.4297e-06     	|
configure_callbacks                	|  7.1768e-06     	|2              	|  1.4354e-05     	|  2.414e-06      	|
on_pretrain_routine_end            	|  1.2856e-05     	|1              	|  1.2856e-05     	|  2.1621e-06     	|
prepare_data                       	|  6.1933e-06     	|2              	|  1.2387e-05     	|  2.0832e-06     	|
on_train_dataloader                	|  8.4192e-06     	|1              	|  8.4192e-06     	|  1.4159e-06     	|
