VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  415.69         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.6831         	|63             	|  358.03         	|  86.129         	|
run_training_batch                 	|  0.33133        	|1000           	|  331.33         	|  79.705         	|
optimizer_step_with_closure_0      	|  0.32866        	|1000           	|  328.66         	|  79.063         	|
training_step_and_backward         	|  0.24601        	|1000           	|  246.01         	|  59.181         	|
model_forward                      	|  0.15322        	|1000           	|  153.22         	|  36.859         	|
training_step                      	|  0.15309        	|1000           	|  153.09         	|  36.827         	|
backward                           	|  0.091479       	|1000           	|  91.479         	|  22.006         	|
run_validate_evaluation            	|  12.479         	|1              	|  12.479         	|  3.002          	|
evaluation_step_and_end            	|  0.24939        	|48             	|  11.97          	|  2.8796         	|
validation_step                    	|  0.24933        	|48             	|  11.968         	|  2.879          	|
get_train_batch                    	|  0.0080987      	|1063           	|  8.6089         	|  2.071          	|
fetch_next_train_batch             	|  0.0080785      	|1063           	|  8.5874         	|  2.0658         	|
zero_grad                          	|  0.0012876      	|1000           	|  1.2876         	|  0.30975        	|
on_train_batch_end                 	|  0.0012041      	|1000           	|  1.2041         	|  0.28966        	|
training_batch_to_device           	|  0.00022779     	|1000           	|  0.22779        	|  0.054798       	|
get_validate_batch                 	|  0.0045916      	|49             	|  0.22499        	|  0.054124       	|
fetch_next_validate_batch          	|  0.0045765      	|49             	|  0.22425        	|  0.053946       	|
on_train_batch_start               	|  9.1913e-05     	|1000           	|  0.091913       	|  0.022111       	|
on_train_end                       	|  0.025653       	|1              	|  0.025653       	|  0.0061711      	|
on_pretrain_routine_start          	|  0.022552       	|1              	|  0.022552       	|  0.0054251      	|
on_train_epoch_start               	|  0.00030573     	|63             	|  0.019261       	|  0.0046335      	|
on_train_epoch_end                 	|  0.00028056     	|63             	|  0.017675       	|  0.0042519      	|
on_batch_start                     	|  1.5801e-05     	|1000           	|  0.015801       	|  0.0038011      	|
on_before_zero_grad                	|  1.3543e-05     	|1000           	|  0.013543       	|  0.003258       	|
on_batch_end                       	|  1.2942e-05     	|1000           	|  0.012942       	|  0.0031133      	|
on_after_backward                  	|  1.1345e-05     	|1000           	|  0.011345       	|  0.0027292      	|
on_before_optimizer_step           	|  1.0573e-05     	|1000           	|  0.010573       	|  0.0025434      	|
on_before_backward                 	|  1.0299e-05     	|1000           	|  0.010299       	|  0.0024776      	|
evaluation_batch_to_device         	|  0.00021296     	|48             	|  0.010222       	|  0.002459       	|
configure_optimizers               	|  0.0081272      	|1              	|  0.0081272      	|  0.0019551      	|
training_step_end                  	|  6.024e-06      	|1000           	|  0.006024       	|  0.0014491      	|
on_validation_model_eval           	|  0.0040288      	|1              	|  0.0040288      	|  0.00096917     	|
on_validation_batch_end            	|  2.1645e-05     	|48             	|  0.001039       	|  0.00024993     	|
on_train_start                     	|  0.00072426     	|1              	|  0.00072426     	|  0.00017423     	|
on_epoch_start                     	|  1.047e-05      	|64             	|  0.00067007     	|  0.00016119     	|
on_epoch_end                       	|  1.0109e-05     	|64             	|  0.00064701     	|  0.00015564     	|
on_validation_batch_start          	|  1.0876e-05     	|48             	|  0.00052204     	|  0.00012558     	|
on_validation_start                	|  0.00032727     	|1              	|  0.00032727     	|  7.8727e-05     	|
validation_step_end                	|  5.3697e-06     	|48             	|  0.00025775     	|  6.2004e-05     	|
on_validation_end                  	|  0.00011629     	|1              	|  0.00011629     	|  2.7975e-05     	|
on_before_accelerator_backend_setup	|  1.7708e-05     	|2              	|  3.5417e-05     	|  8.5199e-06     	|
teardown                           	|  1.5394e-05     	|2              	|  3.0788e-05     	|  7.4064e-06     	|
on_configure_sharded_model         	|  1.347e-05      	|2              	|  2.6941e-05     	|  6.481e-06      	|
setup                              	|  1.3316e-05     	|2              	|  2.6631e-05     	|  6.4064e-06     	|
on_fit_end                         	|  2.2071e-05     	|1              	|  2.2071e-05     	|  5.3094e-06     	|
on_validation_epoch_end            	|  1.8014e-05     	|1              	|  1.8014e-05     	|  4.3335e-06     	|
on_fit_start                       	|  1.559e-05      	|1              	|  1.559e-05      	|  3.7504e-06     	|
on_val_dataloader                  	|  6.1215e-06     	|2              	|  1.2243e-05     	|  2.9452e-06     	|
prepare_data                       	|  5.7911e-06     	|2              	|  1.1582e-05     	|  2.7862e-06     	|
configure_sharded_model            	|  4.9295e-06     	|2              	|  9.8591e-06     	|  2.3717e-06     	|
on_pretrain_routine_end            	|  9.789e-06      	|1              	|  9.789e-06      	|  2.3549e-06     	|
on_validation_epoch_start          	|  9.679e-06      	|1              	|  9.679e-06      	|  2.3284e-06     	|
configure_callbacks                	|  4.7085e-06     	|2              	|  9.417e-06      	|  2.2654e-06     	|
on_train_dataloader                	|  4.127e-06      	|1              	|  4.127e-06      	|  9.9281e-07     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1620.2         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.438         	|63             	|  1539.6         	|  95.025         	|
run_training_batch                 	|  0.18806        	|8000           	|  1504.5         	|  92.856         	|
training_step_and_backward         	|  0.17778        	|8000           	|  1422.3         	|  87.781         	|
model_forward                      	|  0.092541       	|8000           	|  740.32         	|  45.693         	|
training_step                      	|  0.092408       	|8000           	|  739.26         	|  45.627         	|
backward                           	|  0.085056       	|8000           	|  680.45         	|  41.997         	|
optimizer_step_with_closure_0      	|  0.23666        	|1000           	|  236.66         	|  14.607         	|
run_validate_evaluation            	|  37.336         	|1              	|  37.336         	|  2.3044         	|
evaluation_step_and_end            	|  0.096795       	|377            	|  36.492         	|  2.2523         	|
validation_step                    	|  0.096743       	|377            	|  36.472         	|  2.251          	|
get_train_batch                    	|  0.0022612      	|8063           	|  18.232         	|  1.1253         	|
fetch_next_train_batch             	|  0.0022445      	|8063           	|  18.097         	|  1.117          	|
on_train_batch_end                 	|  0.0010793      	|8000           	|  8.6344         	|  0.53291        	|
training_batch_to_device           	|  0.00021568     	|8000           	|  1.7255         	|  0.1065         	|
zero_grad                          	|  0.0013301      	|1000           	|  1.3301         	|  0.08209        	|
on_train_batch_start               	|  9.2671e-05     	|8000           	|  0.74136        	|  0.045757       	|
get_validate_batch                 	|  0.0015754      	|378            	|  0.59549        	|  0.036753       	|
fetch_next_validate_batch          	|  0.0015615      	|378            	|  0.59026        	|  0.036431       	|
on_batch_start                     	|  1.6838e-05     	|8000           	|  0.1347         	|  0.0083137      	|
on_batch_end                       	|  1.5757e-05     	|8000           	|  0.12606        	|  0.0077802      	|
on_before_backward                 	|  1.5265e-05     	|8000           	|  0.12212        	|  0.0075374      	|
on_after_backward                  	|  1.385e-05      	|8000           	|  0.1108         	|  0.0068387      	|
evaluation_batch_to_device         	|  0.00021099     	|377            	|  0.079545       	|  0.0049095      	|
training_step_end                  	|  5.7851e-06     	|8000           	|  0.046281       	|  0.0028565      	|
on_train_end                       	|  0.02538        	|1              	|  0.02538        	|  0.0015664      	|
on_pretrain_routine_start          	|  0.023022       	|1              	|  0.023022       	|  0.0014209      	|
on_train_epoch_start               	|  0.00029403     	|63             	|  0.018524       	|  0.0011433      	|
on_train_epoch_end                 	|  0.00028358     	|63             	|  0.017865       	|  0.0011026      	|
on_before_zero_grad                	|  1.6902e-05     	|1000           	|  0.016902       	|  0.0010432      	|
on_before_optimizer_step           	|  1.3738e-05     	|1000           	|  0.013738       	|  0.0008479      	|
configure_optimizers               	|  0.0081893      	|1              	|  0.0081893      	|  0.00050544     	|
on_validation_batch_end            	|  2.158e-05      	|377            	|  0.0081355      	|  0.00050212     	|
on_validation_batch_start          	|  1.2043e-05     	|377            	|  0.0045402      	|  0.00028022     	|
on_validation_model_eval           	|  0.0041861      	|1              	|  0.0041861      	|  0.00025837     	|
validation_step_end                	|  4.8295e-06     	|377            	|  0.0018207      	|  0.00011237     	|
on_epoch_start                     	|  1.2725e-05     	|64             	|  0.00081441     	|  5.0265e-05     	|
on_epoch_end                       	|  1.2516e-05     	|64             	|  0.00080099     	|  4.9437e-05     	|
on_train_start                     	|  0.00074195     	|1              	|  0.00074195     	|  4.5793e-05     	|
on_validation_start                	|  0.00032764     	|1              	|  0.00032764     	|  2.0222e-05     	|
on_validation_end                  	|  0.00011778     	|1              	|  0.00011778     	|  7.2693e-06     	|
on_before_accelerator_backend_setup	|  2.0919e-05     	|2              	|  4.1838e-05     	|  2.5822e-06     	|
teardown                           	|  1.7813e-05     	|2              	|  3.5627e-05     	|  2.1989e-06     	|
setup                              	|  1.5363e-05     	|2              	|  3.0727e-05     	|  1.8965e-06     	|
on_configure_sharded_model         	|  1.5303e-05     	|2              	|  3.0607e-05     	|  1.8891e-06     	|
on_fit_end                         	|  2.3324e-05     	|1              	|  2.3324e-05     	|  1.4396e-06     	|
on_fit_start                       	|  1.9917e-05     	|1              	|  1.9917e-05     	|  1.2293e-06     	|
on_validation_epoch_end            	|  1.9016e-05     	|1              	|  1.9016e-05     	|  1.1737e-06     	|
on_pretrain_routine_end            	|  1.4617e-05     	|1              	|  1.4617e-05     	|  9.0215e-07     	|
on_val_dataloader                  	|  5.9665e-06     	|2              	|  1.1933e-05     	|  7.365e-07      	|
on_validation_epoch_start          	|  1.1731e-05     	|1              	|  1.1731e-05     	|  7.2403e-07     	|
prepare_data                       	|  5.8254e-06     	|2              	|  1.1651e-05     	|  7.1909e-07     	|
configure_callbacks                	|  5.38e-06       	|2              	|  1.076e-05      	|  6.6411e-07     	|
configure_sharded_model            	|  5.23e-06       	|2              	|  1.046e-05      	|  6.4558e-07     	|
on_train_dataloader                	|  4.5381e-06     	|1              	|  4.5381e-06     	|  2.8009e-07     	|
