VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  507.13         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.5707         	|167            	|  429.3          	|  84.653         	|
run_training_batch                 	|  0.37584        	|1000           	|  375.84         	|  74.11          	|
optimizer_step_with_closure_0      	|  0.37316        	|1000           	|  373.16         	|  73.583         	|
training_step_and_backward         	|  0.29375        	|1000           	|  293.75         	|  57.924         	|
model_forward                      	|  0.17045        	|1000           	|  170.45         	|  33.611         	|
training_step                      	|  0.17031        	|1000           	|  170.31         	|  33.582         	|
backward                           	|  0.12196        	|1000           	|  121.96         	|  24.05          	|
run_validate_evaluation            	|  34.941         	|1              	|  34.941         	|  6.8899         	|
evaluation_step_and_end            	|  0.27475        	|125            	|  34.344         	|  6.7722         	|
validation_step                    	|  0.2747         	|125            	|  34.337         	|  6.7709         	|
get_train_batch                    	|  0.013974       	|1167           	|  16.308         	|  3.2156         	|
fetch_next_train_batch             	|  0.01395        	|1167           	|  16.279         	|  3.2101         	|
zero_grad                          	|  0.0013053      	|1000           	|  1.3053         	|  0.25738        	|
on_train_batch_end                 	|  0.0012317      	|1000           	|  1.2317         	|  0.24288        	|
training_batch_to_device           	|  0.00031721     	|1000           	|  0.31721        	|  0.062549       	|
get_validate_batch                 	|  0.0023417      	|126            	|  0.29506        	|  0.058182       	|
fetch_next_validate_batch          	|  0.0023274      	|126            	|  0.29325        	|  0.057825       	|
on_train_batch_start               	|  9.5601e-05     	|1000           	|  0.095601       	|  0.018851       	|
on_train_epoch_start               	|  0.00028092     	|167            	|  0.046914       	|  0.0092508      	|
on_train_epoch_end                 	|  0.00026192     	|167            	|  0.04374        	|  0.008625       	|
evaluation_batch_to_device         	|  0.00022547     	|125            	|  0.028184       	|  0.0055574      	|
on_train_end                       	|  0.025046       	|1              	|  0.025046       	|  0.0049387      	|
on_pretrain_routine_start          	|  0.022872       	|1              	|  0.022872       	|  0.0045101      	|
on_batch_start                     	|  2.0661e-05     	|1000           	|  0.020661       	|  0.0040741      	|
on_before_zero_grad                	|  1.5715e-05     	|1000           	|  0.015715       	|  0.0030988      	|
on_batch_end                       	|  1.381e-05      	|1000           	|  0.01381        	|  0.0027231      	|
on_after_backward                  	|  1.1848e-05     	|1000           	|  0.011848       	|  0.0023362      	|
on_before_optimizer_step           	|  1.1465e-05     	|1000           	|  0.011465       	|  0.0022607      	|
on_before_backward                 	|  1.0478e-05     	|1000           	|  0.010478       	|  0.0020661      	|
configure_optimizers               	|  0.0081375      	|1              	|  0.0081375      	|  0.0016046      	|
training_step_end                  	|  6.709e-06      	|1000           	|  0.006709       	|  0.0013229      	|
on_validation_model_eval           	|  0.0041623      	|1              	|  0.0041623      	|  0.00082075     	|
on_validation_batch_end            	|  1.9966e-05     	|125            	|  0.0024958      	|  0.00049213     	|
on_epoch_start                     	|  1.0669e-05     	|168            	|  0.0017924      	|  0.00035344     	|
on_epoch_end                       	|  1.0621e-05     	|168            	|  0.0017844      	|  0.00035185     	|
on_validation_batch_start          	|  1.0559e-05     	|125            	|  0.0013199      	|  0.00026026     	|
on_train_start                     	|  0.00069795     	|1              	|  0.00069795     	|  0.00013763     	|
validation_step_end                	|  4.8405e-06     	|125            	|  0.00060506     	|  0.00011931     	|
on_validation_start                	|  0.00034052     	|1              	|  0.00034052     	|  6.7146e-05     	|
on_validation_end                  	|  0.00011375     	|1              	|  0.00011375     	|  2.2431e-05     	|
on_before_accelerator_backend_setup	|  2.181e-05      	|2              	|  4.3621e-05     	|  8.6015e-06     	|
teardown                           	|  1.5594e-05     	|2              	|  3.1189e-05     	|  6.1501e-06     	|
setup                              	|  1.5434e-05     	|2              	|  3.0868e-05     	|  6.0868e-06     	|
on_configure_sharded_model         	|  1.4126e-05     	|2              	|  2.8252e-05     	|  5.5709e-06     	|
on_fit_end                         	|  2.4114e-05     	|1              	|  2.4114e-05     	|  4.755e-06      	|
on_validation_epoch_end            	|  1.9056e-05     	|1              	|  1.9056e-05     	|  3.7576e-06     	|
on_fit_start                       	|  1.7352e-05     	|1              	|  1.7352e-05     	|  3.4216e-06     	|
on_val_dataloader                  	|  6.2719e-06     	|2              	|  1.2544e-05     	|  2.4735e-06     	|
prepare_data                       	|  5.9215e-06     	|2              	|  1.1843e-05     	|  2.3353e-06     	|
configure_sharded_model            	|  5.7255e-06     	|2              	|  1.1451e-05     	|  2.258e-06      	|
configure_callbacks                	|  5.43e-06       	|2              	|  1.086e-05      	|  2.1415e-06     	|
on_pretrain_routine_end            	|  9.8691e-06     	|1              	|  9.8691e-06     	|  1.9461e-06     	|
on_validation_epoch_start          	|  9.258e-06      	|1              	|  9.258e-06      	|  1.8256e-06     	|
on_train_dataloader                	|  4.3779e-06     	|1              	|  4.3779e-06     	|  8.6327e-07     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1695.8         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.287          	|167            	|  1550.9         	|  91.455         	|
run_training_batch                 	|  0.18796        	|8000           	|  1503.7         	|  88.667         	|
training_step_and_backward         	|  0.17777        	|8000           	|  1422.1         	|  83.861         	|
model_forward                      	|  0.093623       	|8000           	|  748.99         	|  44.166         	|
training_step                      	|  0.093489       	|8000           	|  747.91         	|  44.103         	|
backward                           	|  0.083963       	|8000           	|  671.7          	|  39.609         	|
optimizer_step_with_closure_0      	|  0.23619        	|1000           	|  236.19         	|  13.928         	|
run_validate_evaluation            	|  101.32         	|1              	|  101.32         	|  5.9745         	|
evaluation_step_and_end            	|  0.099344       	|1000           	|  99.344         	|  5.8581         	|
validation_step                    	|  0.099294       	|1000           	|  99.294         	|  5.8551         	|
get_train_batch                    	|  0.003342       	|8167           	|  27.294         	|  1.6095         	|
fetch_next_train_batch             	|  0.0033245      	|8167           	|  27.151         	|  1.6011         	|
on_train_batch_end                 	|  0.0010926      	|8000           	|  8.7406         	|  0.51541        	|
training_batch_to_device           	|  0.00022008     	|8000           	|  1.7606         	|  0.10382        	|
get_validate_batch                 	|  0.0014413      	|1001           	|  1.4427         	|  0.085073       	|
fetch_next_validate_batch          	|  0.0014272      	|1001           	|  1.4287         	|  0.084246       	|
zero_grad                          	|  0.0013088      	|1000           	|  1.3088         	|  0.077178       	|
on_train_batch_start               	|  9.2471e-05     	|8000           	|  0.73977        	|  0.043622       	|
evaluation_batch_to_device         	|  0.00020548     	|1000           	|  0.20548        	|  0.012117       	|
on_batch_start                     	|  1.4923e-05     	|8000           	|  0.11938        	|  0.0070397      	|
on_batch_end                       	|  1.3283e-05     	|8000           	|  0.10626        	|  0.006266       	|
on_before_backward                 	|  1.2831e-05     	|8000           	|  0.10265        	|  0.0060531      	|
on_after_backward                  	|  1.1403e-05     	|8000           	|  0.091227       	|  0.0053794      	|
on_train_epoch_start               	|  0.00028982     	|167            	|  0.0484         	|  0.002854       	|
training_step_end                  	|  6.0089e-06     	|8000           	|  0.048071       	|  0.0028346      	|
on_train_epoch_end                 	|  0.00026318     	|167            	|  0.043952       	|  0.0025917      	|
on_train_end                       	|  0.026089       	|1              	|  0.026089       	|  0.0015384      	|
on_pretrain_routine_start          	|  0.02245        	|1              	|  0.02245        	|  0.0013238      	|
on_validation_batch_end            	|  1.8804e-05     	|1000           	|  0.018804       	|  0.0011088      	|
on_before_zero_grad                	|  1.5567e-05     	|1000           	|  0.015567       	|  0.00091795     	|
on_before_optimizer_step           	|  1.2959e-05     	|1000           	|  0.012959       	|  0.00076416     	|
on_validation_batch_start          	|  9.7994e-06     	|1000           	|  0.0097994      	|  0.00057785     	|
configure_optimizers               	|  0.0085191      	|1              	|  0.0085191      	|  0.00050235     	|
validation_step_end                	|  5.174e-06      	|1000           	|  0.005174       	|  0.0003051      	|
on_validation_model_eval           	|  0.0042584      	|1              	|  0.0042584      	|  0.00025111     	|
on_epoch_end                       	|  1.0432e-05     	|168            	|  0.0017526      	|  0.00010335     	|
on_epoch_start                     	|  1.0318e-05     	|168            	|  0.0017334      	|  0.00010222     	|
on_train_start                     	|  0.00073505     	|1              	|  0.00073505     	|  4.3345e-05     	|
on_validation_start                	|  0.00032012     	|1              	|  0.00032012     	|  1.8877e-05     	|
on_validation_end                  	|  0.00011306     	|1              	|  0.00011306     	|  6.6669e-06     	|
on_before_accelerator_backend_setup	|  1.8405e-05     	|2              	|  3.6809e-05     	|  2.1706e-06     	|
teardown                           	|  1.6627e-05     	|2              	|  3.3253e-05     	|  1.9609e-06     	|
setup                              	|  1.4231e-05     	|2              	|  2.8462e-05     	|  1.6783e-06     	|
on_configure_sharded_model         	|  1.3511e-05     	|2              	|  2.7021e-05     	|  1.5934e-06     	|
on_fit_end                         	|  2.2823e-05     	|1              	|  2.2823e-05     	|  1.3458e-06     	|
on_validation_epoch_end            	|  1.8835e-05     	|1              	|  1.8835e-05     	|  1.1107e-06     	|
on_fit_start                       	|  1.7002e-05     	|1              	|  1.7002e-05     	|  1.0026e-06     	|
on_val_dataloader                  	|  6.0614e-06     	|2              	|  1.2123e-05     	|  7.1486e-07     	|
prepare_data                       	|  6.0565e-06     	|2              	|  1.2113e-05     	|  7.1428e-07     	|
on_pretrain_routine_end            	|  1.2113e-05     	|1              	|  1.2113e-05     	|  7.1428e-07     	|
configure_callbacks                	|  5.4603e-06     	|2              	|  1.0921e-05     	|  6.4397e-07     	|
configure_sharded_model            	|  5.0295e-06     	|2              	|  1.0059e-05     	|  5.9316e-07     	|
on_validation_epoch_start          	|  8.947e-06      	|1              	|  8.947e-06      	|  5.2758e-07     	|
on_train_dataloader                	|  4.207e-06      	|1              	|  4.207e-06      	|  2.4808e-07     	|
