VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1653.7         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  71.862         	|21             	|  1509.1         	|  91.253         	|
run_training_batch                 	|  0.18487        	|8000           	|  1479.0         	|  89.433         	|
training_step_and_backward         	|  0.17482        	|8000           	|  1398.5         	|  84.568         	|
model_forward                      	|  0.093561       	|8000           	|  748.48         	|  45.26          	|
training_step                      	|  0.09343        	|8000           	|  747.44         	|  45.197         	|
backward                           	|  0.081077       	|8000           	|  648.61         	|  39.221         	|
optimizer_step_with_closure_0      	|  0.23196        	|1000           	|  231.96         	|  14.026         	|
run_validate_evaluation            	|  101.26         	|1              	|  101.26         	|  6.123          	|
evaluation_step_and_end            	|  0.099257       	|1000           	|  99.257         	|  6.002          	|
validation_step                    	|  0.099207       	|1000           	|  99.207         	|  5.999          	|
get_train_batch                    	|  0.0018379      	|8021           	|  14.742         	|  0.89144        	|
fetch_next_train_batch             	|  0.0018216      	|8021           	|  14.611         	|  0.8835         	|
on_train_batch_end                 	|  0.0010631      	|8000           	|  8.5044         	|  0.51425        	|
training_batch_to_device           	|  0.00020879     	|8000           	|  1.6703         	|  0.101          	|
get_validate_batch                 	|  0.0014692      	|1001           	|  1.4706         	|  0.088928       	|
fetch_next_validate_batch          	|  0.0014551      	|1001           	|  1.4565         	|  0.088074       	|
zero_grad                          	|  0.0012986      	|1000           	|  1.2986         	|  0.078528       	|
on_train_batch_start               	|  8.8984e-05     	|8000           	|  0.71187        	|  0.043046       	|
evaluation_batch_to_device         	|  0.00020499     	|1000           	|  0.20499        	|  0.012396       	|
on_batch_start                     	|  1.4256e-05     	|8000           	|  0.11405        	|  0.0068962      	|
on_batch_end                       	|  1.3514e-05     	|8000           	|  0.10812        	|  0.0065376      	|
on_before_backward                 	|  1.2789e-05     	|8000           	|  0.10231        	|  0.0061865      	|
on_after_backward                  	|  1.1315e-05     	|8000           	|  0.090521       	|  0.0054737      	|
training_step_end                  	|  5.7406e-06     	|8000           	|  0.045925       	|  0.002777       	|
on_train_end                       	|  0.025701       	|1              	|  0.025701       	|  0.0015541      	|
on_pretrain_routine_start          	|  0.023233       	|1              	|  0.023233       	|  0.0014049      	|
on_validation_batch_end            	|  1.7827e-05     	|1000           	|  0.017827       	|  0.001078       	|
on_before_zero_grad                	|  1.3702e-05     	|1000           	|  0.013702       	|  0.00082852     	|
on_before_optimizer_step           	|  1.1201e-05     	|1000           	|  0.011201       	|  0.00067731     	|
on_validation_batch_start          	|  9.887e-06      	|1000           	|  0.009887       	|  0.00059786     	|
configure_optimizers               	|  0.0087271      	|1              	|  0.0087271      	|  0.00052772     	|
on_train_epoch_start               	|  0.00029611     	|21             	|  0.0062184      	|  0.00037602     	|
on_train_epoch_end                 	|  0.00025838     	|21             	|  0.0054261      	|  0.00032811     	|
validation_step_end                	|  4.7148e-06     	|1000           	|  0.0047148      	|  0.0002851      	|
on_validation_model_eval           	|  0.0045678      	|1              	|  0.0045678      	|  0.00027621     	|
on_train_start                     	|  0.00067144     	|1              	|  0.00067144     	|  4.0601e-05     	|
on_validation_start                	|  0.00033216     	|1              	|  0.00033216     	|  2.0085e-05     	|
on_epoch_start                     	|  1.0546e-05     	|22             	|  0.00023202     	|  1.403e-05      	|
on_epoch_end                       	|  9.776e-06      	|22             	|  0.00021507     	|  1.3005e-05     	|
on_validation_end                  	|  0.00011272     	|1              	|  0.00011272     	|  6.8161e-06     	|
on_before_accelerator_backend_setup	|  1.8675e-05     	|2              	|  3.7349e-05     	|  2.2585e-06     	|
teardown                           	|  1.6165e-05     	|2              	|  3.233e-05      	|  1.955e-06      	|
on_configure_sharded_model         	|  1.4848e-05     	|2              	|  2.9696e-05     	|  1.7957e-06     	|
setup                              	|  1.3771e-05     	|2              	|  2.7542e-05     	|  1.6654e-06     	|
on_fit_end                         	|  2.3204e-05     	|1              	|  2.3204e-05     	|  1.4031e-06     	|
on_validation_epoch_end            	|  1.9817e-05     	|1              	|  1.9817e-05     	|  1.1983e-06     	|
on_fit_start                       	|  1.7904e-05     	|1              	|  1.7904e-05     	|  1.0826e-06     	|
configure_callbacks                	|  6.8975e-06     	|2              	|  1.3795e-05     	|  8.3417e-07     	|
prepare_data                       	|  6.452e-06      	|2              	|  1.2904e-05     	|  7.8029e-07     	|
configure_sharded_model            	|  5.9666e-06     	|2              	|  1.1933e-05     	|  7.2159e-07     	|
on_val_dataloader                  	|  5.796e-06      	|2              	|  1.1592e-05     	|  7.0095e-07     	|
on_pretrain_routine_end            	|  9.9291e-06     	|1              	|  9.9291e-06     	|  6.004e-07      	|
on_validation_epoch_start          	|  9.0969e-06     	|1              	|  9.0969e-06     	|  5.5008e-07     	|
on_train_dataloader                	|  4.5991e-06     	|1              	|  4.5991e-06     	|  2.781e-07      	|
