VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  515.9          	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.6217         	|167            	|  437.82         	|  84.864         	|
run_training_batch                 	|  0.38426        	|1000           	|  384.26         	|  74.482         	|
optimizer_step_with_closure_0      	|  0.38164        	|1000           	|  381.64         	|  73.975         	|
training_step_and_backward         	|  0.30109        	|1000           	|  301.09         	|  58.362         	|
model_forward                      	|  0.17412        	|1000           	|  174.12         	|  33.751         	|
training_step                      	|  0.17398        	|1000           	|  173.98         	|  33.723         	|
backward                           	|  0.12564        	|1000           	|  125.64         	|  24.353         	|
run_validate_evaluation            	|  34.961         	|1              	|  34.961         	|  6.7767         	|
evaluation_step_and_end            	|  0.27484        	|125            	|  34.355         	|  6.6593         	|
validation_step                    	|  0.27479        	|125            	|  34.349         	|  6.658          	|
get_train_batch                    	|  0.01399        	|1167           	|  16.326         	|  3.1645         	|
fetch_next_train_batch             	|  0.013965       	|1167           	|  16.297         	|  3.159          	|
zero_grad                          	|  0.0013035      	|1000           	|  1.3035         	|  0.25266        	|
on_train_batch_end                 	|  0.0012415      	|1000           	|  1.2415         	|  0.24065        	|
get_validate_batch                 	|  0.0025556      	|126            	|  0.322          	|  0.062415       	|
fetch_next_validate_batch          	|  0.002541       	|126            	|  0.32016        	|  0.062059       	|
training_batch_to_device           	|  0.0003145      	|1000           	|  0.3145         	|  0.06096        	|
on_train_batch_start               	|  9.5957e-05     	|1000           	|  0.095957       	|  0.0186         	|
on_train_epoch_start               	|  0.00027541     	|167            	|  0.045994       	|  0.0089152      	|
on_train_epoch_end                 	|  0.00026359     	|167            	|  0.04402        	|  0.0085326      	|
evaluation_batch_to_device         	|  0.00022924     	|125            	|  0.028655       	|  0.0055544      	|
on_train_end                       	|  0.025343       	|1              	|  0.025343       	|  0.0049125      	|
on_pretrain_routine_start          	|  0.022892       	|1              	|  0.022892       	|  0.0044373      	|
on_batch_start                     	|  2.0088e-05     	|1000           	|  0.020088       	|  0.0038938      	|
on_before_zero_grad                	|  1.5333e-05     	|1000           	|  0.015333       	|  0.002972       	|
on_batch_end                       	|  1.3854e-05     	|1000           	|  0.013854       	|  0.0026853      	|
on_after_backward                  	|  1.1964e-05     	|1000           	|  0.011964       	|  0.002319       	|
on_before_optimizer_step           	|  1.195e-05      	|1000           	|  0.01195        	|  0.0023164      	|
on_before_backward                 	|  1.0499e-05     	|1000           	|  0.010499       	|  0.0020351      	|
configure_optimizers               	|  0.0081341      	|1              	|  0.0081341      	|  0.0015767      	|
training_step_end                  	|  6.1757e-06     	|1000           	|  0.0061757      	|  0.0011971      	|
on_validation_model_eval           	|  0.0041887      	|1              	|  0.0041887      	|  0.00081191     	|
on_validation_batch_end            	|  1.9615e-05     	|125            	|  0.0024519      	|  0.00047526     	|
on_epoch_start                     	|  1.0858e-05     	|168            	|  0.0018242      	|  0.00035359     	|
on_epoch_end                       	|  1.0583e-05     	|168            	|  0.0017779      	|  0.00034461     	|
on_validation_batch_start          	|  1.0412e-05     	|125            	|  0.0013014      	|  0.00025226     	|
on_train_start                     	|  0.00071938     	|1              	|  0.00071938     	|  0.00013944     	|
validation_step_end                	|  4.9619e-06     	|125            	|  0.00062024     	|  0.00012022     	|
on_validation_start                	|  0.00032567     	|1              	|  0.00032567     	|  6.3126e-05     	|
on_validation_end                  	|  0.00011197     	|1              	|  0.00011197     	|  2.1704e-05     	|
on_before_accelerator_backend_setup	|  1.8951e-05     	|2              	|  3.7902e-05     	|  7.3467e-06     	|
teardown                           	|  1.5434e-05     	|2              	|  3.0867e-05     	|  5.9831e-06     	|
setup                              	|  1.4778e-05     	|2              	|  2.9556e-05     	|  5.729e-06      	|
on_configure_sharded_model         	|  1.376e-05      	|2              	|  2.7521e-05     	|  5.3345e-06     	|
on_validation_epoch_end            	|  2.0529e-05     	|1              	|  2.0529e-05     	|  3.9792e-06     	|
on_fit_end                         	|  1.9717e-05     	|1              	|  1.9717e-05     	|  3.8219e-06     	|
on_fit_start                       	|  1.614e-05      	|1              	|  1.614e-05      	|  3.1285e-06     	|
on_val_dataloader                  	|  6.2315e-06     	|2              	|  1.2463e-05     	|  2.4158e-06     	|
prepare_data                       	|  5.7305e-06     	|2              	|  1.1461e-05     	|  2.2215e-06     	|
configure_callbacks                	|  5.255e-06      	|2              	|  1.051e-05      	|  2.0372e-06     	|
configure_sharded_model            	|  4.869e-06      	|2              	|  9.738e-06      	|  1.8876e-06     	|
on_validation_epoch_start          	|  9.5781e-06     	|1              	|  9.5781e-06     	|  1.8566e-06     	|
on_pretrain_routine_end            	|  9.3071e-06     	|1              	|  9.3071e-06     	|  1.804e-06      	|
on_train_dataloader                	|  3.967e-06      	|1              	|  3.967e-06      	|  7.6894e-07     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1673.6         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.1518         	|167            	|  1528.3         	|  91.323         	|
run_training_batch                 	|  0.18514        	|8000           	|  1481.2         	|  88.503         	|
training_step_and_backward         	|  0.17507        	|8000           	|  1400.6         	|  83.69          	|
model_forward                      	|  0.093972       	|8000           	|  751.78         	|  44.921         	|
training_step                      	|  0.093838       	|8000           	|  750.7          	|  44.857         	|
backward                           	|  0.080917       	|8000           	|  647.34         	|  38.68          	|
optimizer_step_with_closure_0      	|  0.23289        	|1000           	|  232.89         	|  13.916         	|
run_validate_evaluation            	|  101.65         	|1              	|  101.65         	|  6.0739         	|
evaluation_step_and_end            	|  0.099689       	|1000           	|  99.689         	|  5.9567         	|
validation_step                    	|  0.099638       	|1000           	|  99.638         	|  5.9536         	|
get_train_batch                    	|  0.0033512      	|8167           	|  27.369         	|  1.6354         	|
fetch_next_train_batch             	|  0.0033337      	|8167           	|  27.227         	|  1.6269         	|
on_train_batch_end                 	|  0.0010793      	|8000           	|  8.6344         	|  0.51593        	|
training_batch_to_device           	|  0.0002168      	|8000           	|  1.7344         	|  0.10363        	|
get_validate_batch                 	|  0.0014259      	|1001           	|  1.4273         	|  0.085286       	|
fetch_next_validate_batch          	|  0.0014119      	|1001           	|  1.4133         	|  0.084447       	|
zero_grad                          	|  0.0013404      	|1000           	|  1.3404         	|  0.080095       	|
on_train_batch_start               	|  9.1493e-05     	|8000           	|  0.73194        	|  0.043735       	|
evaluation_batch_to_device         	|  0.00020796     	|1000           	|  0.20796        	|  0.012426       	|
on_batch_start                     	|  1.5405e-05     	|8000           	|  0.12324        	|  0.0073641      	|
on_batch_end                       	|  1.3429e-05     	|8000           	|  0.10743        	|  0.0064193      	|
on_before_backward                 	|  1.3324e-05     	|8000           	|  0.10659        	|  0.0063693      	|
on_after_backward                  	|  1.1953e-05     	|8000           	|  0.095623       	|  0.0057137      	|
on_train_epoch_start               	|  0.00028139     	|167            	|  0.046992       	|  0.0028079      	|
training_step_end                  	|  5.7543e-06     	|8000           	|  0.046034       	|  0.0027507      	|
on_train_epoch_end                 	|  0.00026338     	|167            	|  0.043985       	|  0.0026282      	|
on_train_end                       	|  0.025281       	|1              	|  0.025281       	|  0.0015106      	|
on_pretrain_routine_start          	|  0.022466       	|1              	|  0.022466       	|  0.0013424      	|
on_validation_batch_end            	|  1.9749e-05     	|1000           	|  0.019749       	|  0.0011801      	|
on_before_zero_grad                	|  1.5863e-05     	|1000           	|  0.015863       	|  0.00094786     	|
on_before_optimizer_step           	|  1.2839e-05     	|1000           	|  0.012839       	|  0.00076715     	|
on_validation_batch_start          	|  9.5872e-06     	|1000           	|  0.0095872      	|  0.00057286     	|
configure_optimizers               	|  0.0082084      	|1              	|  0.0082084      	|  0.00049047     	|
validation_step_end                	|  4.6497e-06     	|1000           	|  0.0046497      	|  0.00027783     	|
on_validation_model_eval           	|  0.0040219      	|1              	|  0.0040219      	|  0.00024032     	|
on_epoch_start                     	|  1.1216e-05     	|168            	|  0.0018843      	|  0.00011259     	|
on_epoch_end                       	|  1.1088e-05     	|168            	|  0.0018628      	|  0.00011131     	|
on_train_start                     	|  0.0007151      	|1              	|  0.0007151      	|  4.2729e-05     	|
on_validation_start                	|  0.00031531     	|1              	|  0.00031531     	|  1.8841e-05     	|
on_validation_end                  	|  0.00010452     	|1              	|  0.00010452     	|  6.2456e-06     	|
on_before_accelerator_backend_setup	|  1.9151e-05     	|2              	|  3.8301e-05     	|  2.2886e-06     	|
teardown                           	|  1.5574e-05     	|2              	|  3.1149e-05     	|  1.8612e-06     	|
setup                              	|  1.3892e-05     	|2              	|  2.7783e-05     	|  1.6601e-06     	|
on_configure_sharded_model         	|  1.354e-05      	|2              	|  2.708e-05      	|  1.6181e-06     	|
on_fit_end                         	|  2.128e-05      	|1              	|  2.128e-05      	|  1.2715e-06     	|
on_validation_epoch_end            	|  1.8044e-05     	|1              	|  1.8044e-05     	|  1.0782e-06     	|
on_fit_start                       	|  1.6752e-05     	|1              	|  1.6752e-05     	|  1.001e-06      	|
on_val_dataloader                  	|  6.3016e-06     	|2              	|  1.2603e-05     	|  7.5307e-07     	|
on_pretrain_routine_end            	|  1.2092e-05     	|1              	|  1.2092e-05     	|  7.2253e-07     	|
configure_callbacks                	|  5.8755e-06     	|2              	|  1.1751e-05     	|  7.0215e-07     	|
prepare_data                       	|  5.8359e-06     	|2              	|  1.1672e-05     	|  6.9742e-07     	|
configure_sharded_model            	|  5.2096e-06     	|2              	|  1.0419e-05     	|  6.2257e-07     	|
on_validation_epoch_start          	|  9.3479e-06     	|1              	|  9.3479e-06     	|  5.5856e-07     	|
on_train_dataloader                	|  3.8571e-06     	|1              	|  3.8571e-06     	|  2.3047e-07     	|
