VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  431.85         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.9431         	|63             	|  374.41         	|  86.7           	|
run_training_batch                 	|  0.34767        	|1000           	|  347.67         	|  80.509         	|
optimizer_step_with_closure_0      	|  0.34503        	|1000           	|  345.03         	|  79.897         	|
training_step_and_backward         	|  0.25887        	|1000           	|  258.87         	|  59.946         	|
model_forward                      	|  0.16027        	|1000           	|  160.27         	|  37.112         	|
training_step                      	|  0.16013        	|1000           	|  160.13         	|  37.08          	|
backward                           	|  0.097261       	|1000           	|  97.261         	|  22.522         	|
run_validate_evaluation            	|  12.214         	|1              	|  12.214         	|  2.8284         	|
evaluation_step_and_end            	|  0.24388        	|48             	|  11.706         	|  2.7108         	|
validation_step                    	|  0.24383        	|48             	|  11.704         	|  2.7102         	|
get_train_batch                    	|  0.0080992      	|1063           	|  8.6094         	|  1.9936         	|
fetch_next_train_batch             	|  0.0080802      	|1063           	|  8.5893         	|  1.989          	|
zero_grad                          	|  0.0013206      	|1000           	|  1.3206         	|  0.30581        	|
on_train_batch_end                 	|  0.0012148      	|1000           	|  1.2148         	|  0.28131        	|
training_batch_to_device           	|  0.00022627     	|1000           	|  0.22627        	|  0.052396       	|
get_validate_batch                 	|  0.0046144      	|49             	|  0.2261         	|  0.052358       	|
fetch_next_validate_batch          	|  0.0045993      	|49             	|  0.22537        	|  0.052187       	|
on_train_batch_start               	|  9.1406e-05     	|1000           	|  0.091406       	|  0.021166       	|
on_train_end                       	|  0.025662       	|1              	|  0.025662       	|  0.0059424      	|
on_pretrain_routine_start          	|  0.022795       	|1              	|  0.022795       	|  0.0052786      	|
on_train_epoch_start               	|  0.00029629     	|63             	|  0.018666       	|  0.0043225      	|
on_train_epoch_end                 	|  0.00028201     	|63             	|  0.017767       	|  0.0041141      	|
on_batch_start                     	|  1.6257e-05     	|1000           	|  0.016257       	|  0.0037645      	|
on_before_zero_grad                	|  1.4376e-05     	|1000           	|  0.014376       	|  0.0033289      	|
on_batch_end                       	|  1.3149e-05     	|1000           	|  0.013149       	|  0.0030448      	|
on_after_backward                  	|  1.1734e-05     	|1000           	|  0.011734       	|  0.0027172      	|
evaluation_batch_to_device         	|  0.00021891     	|48             	|  0.010508       	|  0.0024332      	|
on_before_optimizer_step           	|  1.0407e-05     	|1000           	|  0.010407       	|  0.0024098      	|
on_before_backward                 	|  1.0133e-05     	|1000           	|  0.010133       	|  0.0023465      	|
configure_optimizers               	|  0.0086511      	|1              	|  0.0086511      	|  0.0020033      	|
training_step_end                  	|  5.99e-06       	|1000           	|  0.00599        	|  0.0013871      	|
on_validation_model_eval           	|  0.004471       	|1              	|  0.004471       	|  0.0010353      	|
on_validation_batch_end            	|  1.9956e-05     	|48             	|  0.0009579      	|  0.00022181     	|
on_train_start                     	|  0.00076707     	|1              	|  0.00076707     	|  0.00017763     	|
on_epoch_start                     	|  1.0268e-05     	|64             	|  0.00065714     	|  0.00015217     	|
on_epoch_end                       	|  9.9283e-06     	|64             	|  0.00063541     	|  0.00014714     	|
on_validation_batch_start          	|  9.9975e-06     	|48             	|  0.00047988     	|  0.00011112     	|
on_validation_start                	|  0.00032372     	|1              	|  0.00032372     	|  7.4963e-05     	|
validation_step_end                	|  5.0476e-06     	|48             	|  0.00024228     	|  5.6104e-05     	|
on_validation_end                  	|  0.00011739     	|1              	|  0.00011739     	|  2.7183e-05     	|
on_before_accelerator_backend_setup	|  1.845e-05      	|2              	|  3.6899e-05     	|  8.5445e-06     	|
teardown                           	|  1.4141e-05     	|2              	|  2.8283e-05     	|  6.5493e-06     	|
on_configure_sharded_model         	|  1.4021e-05     	|2              	|  2.8042e-05     	|  6.4935e-06     	|
setup                              	|  1.3034e-05     	|2              	|  2.6069e-05     	|  6.0366e-06     	|
on_validation_epoch_end            	|  2.1981e-05     	|1              	|  2.1981e-05     	|  5.09e-06       	|
on_fit_end                         	|  2.1861e-05     	|1              	|  2.1861e-05     	|  5.0622e-06     	|
on_fit_start                       	|  1.7182e-05     	|1              	|  1.7182e-05     	|  3.9787e-06     	|
prepare_data                       	|  5.6661e-06     	|2              	|  1.1332e-05     	|  2.6241e-06     	|
on_val_dataloader                  	|  5.5e-06        	|2              	|  1.1e-05        	|  2.5472e-06     	|
configure_sharded_model            	|  5.1495e-06     	|2              	|  1.0299e-05     	|  2.3849e-06     	|
on_pretrain_routine_end            	|  9.939e-06      	|1              	|  9.939e-06      	|  2.3015e-06     	|
on_validation_epoch_start          	|  9.587e-06      	|1              	|  9.587e-06      	|  2.22e-06       	|
configure_callbacks                	|  4.5585e-06     	|2              	|  9.1171e-06     	|  2.1112e-06     	|
on_train_dataloader                	|  4.539e-06      	|1              	|  4.539e-06      	|  1.0511e-06     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1600.6         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.123         	|63             	|  1519.7         	|  94.949         	|
run_training_batch                 	|  0.18556        	|8000           	|  1484.5         	|  92.747         	|
training_step_and_backward         	|  0.17545        	|8000           	|  1403.6         	|  87.695         	|
model_forward                      	|  0.092764       	|8000           	|  742.11         	|  46.365         	|
training_step                      	|  0.092633       	|8000           	|  741.07         	|  46.299         	|
backward                           	|  0.082511       	|8000           	|  660.09         	|  41.24          	|
optimizer_step_with_closure_0      	|  0.23328        	|1000           	|  233.28         	|  14.575         	|
run_validate_evaluation            	|  37.383         	|1              	|  37.383         	|  2.3356         	|
evaluation_step_and_end            	|  0.09689        	|377            	|  36.527         	|  2.2821         	|
validation_step                    	|  0.096838       	|377            	|  36.508         	|  2.2809         	|
get_train_batch                    	|  0.0022773      	|8063           	|  18.362         	|  1.1472         	|
fetch_next_train_batch             	|  0.0022606      	|8063           	|  18.227         	|  1.1388         	|
on_train_batch_end                 	|  0.0010813      	|8000           	|  8.6502         	|  0.54044        	|
training_batch_to_device           	|  0.00021532     	|8000           	|  1.7226         	|  0.10762        	|
zero_grad                          	|  0.0012879      	|1000           	|  1.2879         	|  0.080465       	|
on_train_batch_start               	|  9.1231e-05     	|8000           	|  0.72984        	|  0.045598       	|
get_validate_batch                 	|  0.0016087      	|378            	|  0.60808        	|  0.037991       	|
fetch_next_validate_batch          	|  0.0015949      	|378            	|  0.60288        	|  0.037666       	|
on_batch_start                     	|  1.5441e-05     	|8000           	|  0.12353        	|  0.0077179      	|
on_batch_end                       	|  1.3119e-05     	|8000           	|  0.10495        	|  0.0065569      	|
on_before_backward                 	|  1.3016e-05     	|8000           	|  0.10412        	|  0.0065054      	|
on_after_backward                  	|  1.1561e-05     	|8000           	|  0.092489       	|  0.0057784      	|
evaluation_batch_to_device         	|  0.00020841     	|377            	|  0.078569       	|  0.0049087      	|
training_step_end                  	|  5.9023e-06     	|8000           	|  0.047218       	|  0.0029501      	|
on_train_end                       	|  0.025262       	|1              	|  0.025262       	|  0.0015783      	|
on_pretrain_routine_start          	|  0.022707       	|1              	|  0.022707       	|  0.0014186      	|
on_train_epoch_start               	|  0.00032303     	|63             	|  0.020351       	|  0.0012715      	|
on_train_epoch_end                 	|  0.00028086     	|63             	|  0.017694       	|  0.0011055      	|
on_before_zero_grad                	|  1.4534e-05     	|1000           	|  0.014534       	|  0.00090806     	|
on_before_optimizer_step           	|  1.2189e-05     	|1000           	|  0.012189       	|  0.00076155     	|
configure_optimizers               	|  0.0088455      	|1              	|  0.0088455      	|  0.00055264     	|
on_validation_batch_end            	|  1.8587e-05     	|377            	|  0.0070073      	|  0.0004378      	|
on_validation_model_eval           	|  0.0039782      	|1              	|  0.0039782      	|  0.00024855     	|
on_validation_batch_start          	|  1.0082e-05     	|377            	|  0.003801       	|  0.00023748     	|
validation_step_end                	|  4.9799e-06     	|377            	|  0.0018774      	|  0.00011729     	|
on_train_start                     	|  0.00074918     	|1              	|  0.00074918     	|  4.6807e-05     	|
on_epoch_start                     	|  1.0754e-05     	|64             	|  0.00068828     	|  4.3002e-05     	|
on_epoch_end                       	|  1.056e-05      	|64             	|  0.00067582     	|  4.2223e-05     	|
on_validation_start                	|  0.00032253     	|1              	|  0.00032253     	|  2.0151e-05     	|
on_validation_end                  	|  0.00011178     	|1              	|  0.00011178     	|  6.9836e-06     	|
on_before_accelerator_backend_setup	|  2.0694e-05     	|2              	|  4.1388e-05     	|  2.5858e-06     	|
teardown                           	|  1.865e-05      	|2              	|  3.73e-05       	|  2.3304e-06     	|
on_configure_sharded_model         	|  1.4603e-05     	|2              	|  2.9206e-05     	|  1.8247e-06     	|
setup                              	|  1.37e-05       	|2              	|  2.7401e-05     	|  1.7119e-06     	|
on_fit_end                         	|  2.2902e-05     	|1              	|  2.2902e-05     	|  1.4308e-06     	|
on_fit_start                       	|  1.9327e-05     	|1              	|  1.9327e-05     	|  1.2075e-06     	|
on_validation_epoch_end            	|  1.8344e-05     	|1              	|  1.8344e-05     	|  1.1461e-06     	|
prepare_data                       	|  7.0835e-06     	|2              	|  1.4167e-05     	|  8.8511e-07     	|
on_val_dataloader                  	|  6.407e-06      	|2              	|  1.2814e-05     	|  8.0058e-07     	|
configure_callbacks                	|  5.9016e-06     	|2              	|  1.1803e-05     	|  7.3742e-07     	|
on_pretrain_routine_end            	|  1.1321e-05     	|1              	|  1.1321e-05     	|  7.0731e-07     	|
configure_sharded_model            	|  5.2499e-06     	|2              	|  1.05e-05       	|  6.5599e-07     	|
on_validation_epoch_start          	|  9.598e-06      	|1              	|  9.598e-06      	|  5.9965e-07     	|
on_train_dataloader                	|  4.1679e-06     	|1              	|  4.1679e-06     	|  2.604e-07      	|
