VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1862.5         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  26.696         	|63             	|  1681.9         	|  90.301         	|
run_training_batch                 	|  0.20583        	|8000           	|  1646.7         	|  88.412         	|
training_step_and_backward         	|  0.19432        	|8000           	|  1554.5         	|  83.465         	|
model_forward                      	|  0.10854        	|8000           	|  868.33         	|  46.622         	|
training_step                      	|  0.10841        	|8000           	|  867.27         	|  46.565         	|
backward                           	|  0.085585       	|8000           	|  684.68         	|  36.762         	|
optimizer_step_with_closure_0      	|  0.25548        	|1000           	|  255.48         	|  13.717         	|
run_validate_evaluation            	|  137.77         	|1              	|  137.77         	|  7.3973         	|
evaluation_step_and_end            	|  0.13576        	|1000           	|  135.76         	|  7.2894         	|
validation_step                    	|  0.13571        	|1000           	|  135.71         	|  7.2866         	|
get_train_batch                    	|  0.0022857      	|8063           	|  18.429         	|  0.98951        	|
fetch_next_train_batch             	|  0.002269       	|8063           	|  18.295         	|  0.98227        	|
on_train_batch_end                 	|  0.0010845      	|8000           	|  8.6761         	|  0.46583        	|
training_batch_to_device           	|  0.00021696     	|8000           	|  1.7357         	|  0.09319        	|
get_validate_batch                 	|  0.0014765      	|1001           	|  1.478          	|  0.079356       	|
fetch_next_validate_batch          	|  0.001463       	|1001           	|  1.4644         	|  0.078628       	|
zero_grad                          	|  0.0013675      	|1000           	|  1.3675         	|  0.073421       	|
on_train_batch_start               	|  9.1185e-05     	|8000           	|  0.72948        	|  0.039167       	|
evaluation_batch_to_device         	|  0.0002097      	|1000           	|  0.2097         	|  0.011259       	|
on_batch_start                     	|  1.4676e-05     	|8000           	|  0.11741        	|  0.0063037      	|
on_batch_end                       	|  1.2633e-05     	|8000           	|  0.10106        	|  0.0054262      	|
on_before_backward                 	|  1.2171e-05     	|8000           	|  0.097372       	|  0.005228       	|
on_after_backward                  	|  1.0856e-05     	|8000           	|  0.086849       	|  0.0046631      	|
training_step_end                  	|  5.9959e-06     	|8000           	|  0.047967       	|  0.0025754      	|
on_train_end                       	|  0.025987       	|1              	|  0.025987       	|  0.0013953      	|
on_pretrain_routine_start          	|  0.022725       	|1              	|  0.022725       	|  0.0012201      	|
on_train_epoch_start               	|  0.00030343     	|63             	|  0.019116       	|  0.0010264      	|
on_validation_batch_end            	|  1.8552e-05     	|1000           	|  0.018552       	|  0.00099609     	|
on_train_epoch_end                 	|  0.00028201     	|63             	|  0.017766       	|  0.00095391     	|
on_before_zero_grad                	|  1.4673e-05     	|1000           	|  0.014673       	|  0.00078782     	|
on_before_optimizer_step           	|  1.1433e-05     	|1000           	|  0.011433       	|  0.00061383     	|
on_validation_batch_start          	|  9.5675e-06     	|1000           	|  0.0095675      	|  0.00051369     	|
configure_optimizers               	|  0.0082228      	|1              	|  0.0082228      	|  0.0004415      	|
validation_step_end                	|  4.638e-06      	|1000           	|  0.004638       	|  0.00024902     	|
on_validation_model_eval           	|  0.0041631      	|1              	|  0.0041631      	|  0.00022352     	|
on_train_start                     	|  0.00080513     	|1              	|  0.00080513     	|  4.3229e-05     	|
on_epoch_start                     	|  1.0894e-05     	|64             	|  0.00069725     	|  3.7436e-05     	|
on_epoch_end                       	|  1.0496e-05     	|64             	|  0.00067173     	|  3.6067e-05     	|
on_validation_start                	|  0.00032099     	|1              	|  0.00032099     	|  1.7235e-05     	|
on_validation_end                  	|  0.00011335     	|1              	|  0.00011335     	|  6.0862e-06     	|
setup                              	|  2.5257e-05     	|2              	|  5.0514e-05     	|  2.7122e-06     	|
on_before_accelerator_backend_setup	|  1.7984e-05     	|2              	|  3.5968e-05     	|  1.9312e-06     	|
on_configure_sharded_model         	|  1.7508e-05     	|2              	|  3.5016e-05     	|  1.8801e-06     	|
teardown                           	|  1.4658e-05     	|2              	|  2.9315e-05     	|  1.574e-06      	|
on_fit_end                         	|  2.0298e-05     	|1              	|  2.0298e-05     	|  1.0898e-06     	|
on_validation_epoch_end            	|  1.7944e-05     	|1              	|  1.7944e-05     	|  9.6344e-07     	|
on_fit_start                       	|  1.6852e-05     	|1              	|  1.6852e-05     	|  9.0481e-07     	|
prepare_data                       	|  6.2421e-06     	|2              	|  1.2484e-05     	|  6.7029e-07     	|
on_val_dataloader                  	|  6.2216e-06     	|2              	|  1.2443e-05     	|  6.6809e-07     	|
configure_sharded_model            	|  5.8105e-06     	|2              	|  1.1621e-05     	|  6.2395e-07     	|
on_pretrain_routine_end            	|  1.0108e-05     	|1              	|  1.0108e-05     	|  5.4272e-07     	|
on_validation_epoch_start          	|  9.0969e-06     	|1              	|  9.0969e-06     	|  4.8843e-07     	|
configure_callbacks                	|  4.4685e-06     	|2              	|  8.937e-06      	|  4.7984e-07     	|
on_train_dataloader                	|  4.017e-06      	|1              	|  4.017e-06      	|  2.1568e-07     	|
