VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  411.61         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.62           	|63             	|  354.06         	|  86.019         	|
run_training_batch                 	|  0.32746        	|1000           	|  327.46         	|  79.555         	|
optimizer_step_with_closure_0      	|  0.32478        	|1000           	|  324.78         	|  78.905         	|
training_step_and_backward         	|  0.24294        	|1000           	|  242.94         	|  59.021         	|
model_forward                      	|  0.15094        	|1000           	|  150.94         	|  36.672         	|
training_step                      	|  0.15081        	|1000           	|  150.81         	|  36.639         	|
backward                           	|  0.090679       	|1000           	|  90.679         	|  22.03          	|
run_validate_evaluation            	|  12.334         	|1              	|  12.334         	|  2.9967         	|
evaluation_step_and_end            	|  0.24673        	|48             	|  11.843         	|  2.8772         	|
validation_step                    	|  0.24668        	|48             	|  11.84          	|  2.8766         	|
get_train_batch                    	|  0.0080677      	|1063           	|  8.5759         	|  2.0835         	|
fetch_next_train_batch             	|  0.0080488      	|1063           	|  8.5559         	|  2.0786         	|
zero_grad                          	|  0.0012861      	|1000           	|  1.2861         	|  0.31246        	|
on_train_batch_end                 	|  0.0011872      	|1000           	|  1.1872         	|  0.28844        	|
training_batch_to_device           	|  0.00022583     	|1000           	|  0.22583        	|  0.054865       	|
get_validate_batch                 	|  0.0042618      	|49             	|  0.20883        	|  0.050735       	|
fetch_next_validate_batch          	|  0.0042469      	|49             	|  0.2081         	|  0.050557       	|
on_train_batch_start               	|  9.2926e-05     	|1000           	|  0.092926       	|  0.022576       	|
on_train_end                       	|  0.025544       	|1              	|  0.025544       	|  0.0062058      	|
on_pretrain_routine_start          	|  0.02259        	|1              	|  0.02259        	|  0.0054882      	|
on_train_epoch_start               	|  0.00028696     	|63             	|  0.018079       	|  0.0043922      	|
on_train_epoch_end                 	|  0.00028036     	|63             	|  0.017663       	|  0.0042911      	|
on_batch_start                     	|  1.6988e-05     	|1000           	|  0.016988       	|  0.0041273      	|
on_before_zero_grad                	|  1.4095e-05     	|1000           	|  0.014095       	|  0.0034243      	|
on_batch_end                       	|  1.3284e-05     	|1000           	|  0.013284       	|  0.0032272      	|
on_before_optimizer_step           	|  1.1251e-05     	|1000           	|  0.011251       	|  0.0027335      	|
on_after_backward                  	|  1.094e-05      	|1000           	|  0.01094        	|  0.0026579      	|
on_before_backward                 	|  1.0436e-05     	|1000           	|  0.010436       	|  0.0025353      	|
evaluation_batch_to_device         	|  0.00021669     	|48             	|  0.010401       	|  0.0025269      	|
configure_optimizers               	|  0.008794       	|1              	|  0.008794       	|  0.0021365      	|
training_step_end                  	|  5.9699e-06     	|1000           	|  0.0059699      	|  0.0014504      	|
on_validation_model_eval           	|  0.0040866      	|1              	|  0.0040866      	|  0.00099284     	|
on_validation_batch_end            	|  2.0457e-05     	|48             	|  0.00098192     	|  0.00023856     	|
on_train_start                     	|  0.00074638     	|1              	|  0.00074638     	|  0.00018133     	|
on_epoch_start                     	|  1.0713e-05     	|64             	|  0.0006856      	|  0.00016657     	|
on_epoch_end                       	|  1.0209e-05     	|64             	|  0.00065341     	|  0.00015874     	|
on_validation_batch_start          	|  1.143e-05      	|48             	|  0.00054864     	|  0.00013329     	|
on_validation_start                	|  0.00032176     	|1              	|  0.00032176     	|  7.8172e-05     	|
validation_step_end                	|  5.1894e-06     	|48             	|  0.00024909     	|  6.0516e-05     	|
on_validation_end                  	|  0.00011192     	|1              	|  0.00011192     	|  2.7191e-05     	|
on_before_accelerator_backend_setup	|  1.868e-05      	|2              	|  3.736e-05      	|  9.0766e-06     	|
teardown                           	|  1.4266e-05     	|2              	|  2.8533e-05     	|  6.9321e-06     	|
on_configure_sharded_model         	|  1.4146e-05     	|2              	|  2.8292e-05     	|  6.8735e-06     	|
setup                              	|  1.3064e-05     	|2              	|  2.6129e-05     	|  6.348e-06      	|
on_fit_end                         	|  2.4206e-05     	|1              	|  2.4206e-05     	|  5.8808e-06     	|
on_validation_epoch_end            	|  1.8835e-05     	|1              	|  1.8835e-05     	|  4.5759e-06     	|
on_fit_start                       	|  1.7122e-05     	|1              	|  1.7122e-05     	|  4.1598e-06     	|
on_val_dataloader                  	|  6.1515e-06     	|2              	|  1.2303e-05     	|  2.989e-06      	|
prepare_data                       	|  6.0415e-06     	|2              	|  1.2083e-05     	|  2.9356e-06     	|
configure_sharded_model            	|  4.9496e-06     	|2              	|  9.8991e-06     	|  2.405e-06      	|
on_pretrain_routine_end            	|  9.858e-06      	|1              	|  9.858e-06      	|  2.395e-06      	|
configure_callbacks                	|  4.8795e-06     	|2              	|  9.759e-06      	|  2.3709e-06     	|
on_validation_epoch_start          	|  9.177e-06      	|1              	|  9.177e-06      	|  2.2296e-06     	|
on_train_dataloader                	|  4.157e-06      	|1              	|  4.157e-06      	|  1.0099e-06     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1605.1         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.201         	|63             	|  1524.7         	|  94.989         	|
run_training_batch                 	|  0.1862         	|8000           	|  1489.6         	|  92.802         	|
training_step_and_backward         	|  0.17611        	|8000           	|  1408.9         	|  87.776         	|
model_forward                      	|  0.092107       	|8000           	|  736.86         	|  45.907         	|
training_step                      	|  0.091976       	|8000           	|  735.81         	|  45.841         	|
backward                           	|  0.083823       	|8000           	|  670.59         	|  41.778         	|
optimizer_step_with_closure_0      	|  0.23454        	|1000           	|  234.54         	|  14.612         	|
run_validate_evaluation            	|  37.267         	|1              	|  37.267         	|  2.3218         	|
evaluation_step_and_end            	|  0.096595       	|377            	|  36.416         	|  2.2688         	|
validation_step                    	|  0.096544       	|377            	|  36.397         	|  2.2676         	|
get_train_batch                    	|  0.0022751      	|8063           	|  18.345         	|  1.1429         	|
fetch_next_train_batch             	|  0.0022589      	|8063           	|  18.214         	|  1.1347         	|
on_train_batch_end                 	|  0.0010797      	|8000           	|  8.6375         	|  0.53812        	|
training_batch_to_device           	|  0.00021284     	|8000           	|  1.7027         	|  0.10608        	|
zero_grad                          	|  0.001309       	|1000           	|  1.309          	|  0.08155        	|
on_train_batch_start               	|  9.0894e-05     	|8000           	|  0.72715        	|  0.045302       	|
get_validate_batch                 	|  0.001599       	|378            	|  0.60443        	|  0.037657       	|
fetch_next_validate_batch          	|  0.0015855      	|378            	|  0.5993         	|  0.037337       	|
on_batch_start                     	|  1.5003e-05     	|8000           	|  0.12002        	|  0.0074774      	|
on_batch_end                       	|  1.3253e-05     	|8000           	|  0.10602        	|  0.0066053      	|
on_before_backward                 	|  1.2387e-05     	|8000           	|  0.099098       	|  0.0061738      	|
on_after_backward                  	|  1.1618e-05     	|8000           	|  0.09294        	|  0.0057902      	|
evaluation_batch_to_device         	|  0.00021092     	|377            	|  0.079516       	|  0.0049539      	|
training_step_end                  	|  5.5882e-06     	|8000           	|  0.044706       	|  0.0027852      	|
on_train_end                       	|  0.025278       	|1              	|  0.025278       	|  0.0015749      	|
on_pretrain_routine_start          	|  0.023052       	|1              	|  0.023052       	|  0.0014362      	|
on_train_epoch_start               	|  0.00029132     	|63             	|  0.018353       	|  0.0011434      	|
on_train_epoch_end                 	|  0.00027538     	|63             	|  0.017349       	|  0.0010809      	|
on_before_zero_grad                	|  1.4479e-05     	|1000           	|  0.014479       	|  0.00090203     	|
on_before_optimizer_step           	|  1.233e-05      	|1000           	|  0.01233        	|  0.00076815     	|
configure_optimizers               	|  0.0082127      	|1              	|  0.0082127      	|  0.00051166     	|
on_validation_batch_end            	|  1.8622e-05     	|377            	|  0.0070205      	|  0.00043738     	|
on_validation_model_eval           	|  0.0040626      	|1              	|  0.0040626      	|  0.0002531      	|
on_validation_batch_start          	|  9.7069e-06     	|377            	|  0.0036595      	|  0.00022799     	|
validation_step_end                	|  5.1219e-06     	|377            	|  0.001931       	|  0.0001203      	|
on_train_start                     	|  0.00073843     	|1              	|  0.00073843     	|  4.6005e-05     	|
on_epoch_end                       	|  1.0836e-05     	|64             	|  0.00069353     	|  4.3207e-05     	|
on_epoch_start                     	|  1.0484e-05     	|64             	|  0.00067095     	|  4.18e-05       	|
on_validation_start                	|  0.00031956     	|1              	|  0.00031956     	|  1.9909e-05     	|
on_validation_end                  	|  0.00012491     	|1              	|  0.00012491     	|  7.7822e-06     	|
on_before_accelerator_backend_setup	|  1.8985e-05     	|2              	|  3.797e-05      	|  2.3656e-06     	|
teardown                           	|  1.5159e-05     	|2              	|  3.0317e-05     	|  1.8888e-06     	|
setup                              	|  1.4307e-05     	|2              	|  2.8614e-05     	|  1.7827e-06     	|
on_configure_sharded_model         	|  1.3881e-05     	|2              	|  2.7762e-05     	|  1.7296e-06     	|
on_fit_end                         	|  2.1349e-05     	|1              	|  2.1349e-05     	|  1.3301e-06     	|
on_validation_epoch_end            	|  1.8084e-05     	|1              	|  1.8084e-05     	|  1.1266e-06     	|
on_fit_start                       	|  1.6902e-05     	|1              	|  1.6902e-05     	|  1.053e-06      	|
on_pretrain_routine_end            	|  1.2674e-05     	|1              	|  1.2674e-05     	|  7.8959e-07     	|
on_val_dataloader                  	|  6.0014e-06     	|2              	|  1.2003e-05     	|  7.4779e-07     	|
prepare_data                       	|  5.9914e-06     	|2              	|  1.1983e-05     	|  7.4654e-07     	|
configure_callbacks                	|  5.2355e-06     	|2              	|  1.0471e-05     	|  6.5236e-07     	|
configure_sharded_model            	|  4.8541e-06     	|2              	|  9.7081e-06     	|  6.0482e-07     	|
on_validation_epoch_start          	|  9.2671e-06     	|1              	|  9.2671e-06     	|  5.7735e-07     	|
on_train_dataloader                	|  3.8971e-06     	|1              	|  3.8971e-06     	|  2.4279e-07     	|
