VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  516.5          	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.6239         	|167            	|  438.19         	|  84.839         	|
run_training_batch                 	|  0.38511        	|1000           	|  385.11         	|  74.562         	|
optimizer_step_with_closure_0      	|  0.38242        	|1000           	|  382.42         	|  74.04          	|
training_step_and_backward         	|  0.30094        	|1000           	|  300.94         	|  58.265         	|
model_forward                      	|  0.17402        	|1000           	|  174.02         	|  33.693         	|
training_step                      	|  0.17388        	|1000           	|  173.88         	|  33.665         	|
backward                           	|  0.12557        	|1000           	|  125.57         	|  24.312         	|
run_validate_evaluation            	|  35.02          	|1              	|  35.02          	|  6.7803         	|
evaluation_step_and_end            	|  0.27538        	|125            	|  34.423         	|  6.6647         	|
validation_step                    	|  0.27533        	|125            	|  34.417         	|  6.6634         	|
get_train_batch                    	|  0.013937       	|1167           	|  16.264         	|  3.1489         	|
fetch_next_train_batch             	|  0.013914       	|1167           	|  16.238         	|  3.1438         	|
zero_grad                          	|  0.0013157      	|1000           	|  1.3157         	|  0.25473        	|
on_train_batch_end                 	|  0.0012188      	|1000           	|  1.2188         	|  0.23597        	|
get_validate_batch                 	|  0.0025563      	|126            	|  0.3221         	|  0.062362       	|
training_batch_to_device           	|  0.00032178     	|1000           	|  0.32178        	|  0.0623         	|
fetch_next_validate_batch          	|  0.002542       	|126            	|  0.3203         	|  0.062013       	|
on_train_batch_start               	|  9.5575e-05     	|1000           	|  0.095575       	|  0.018504       	|
on_train_epoch_end                 	|  0.00027395     	|167            	|  0.045749       	|  0.0088575      	|
on_train_epoch_start               	|  0.00027008     	|167            	|  0.045104       	|  0.0087326      	|
evaluation_batch_to_device         	|  0.00022611     	|125            	|  0.028264       	|  0.0054722      	|
on_train_end                       	|  0.025268       	|1              	|  0.025268       	|  0.0048921      	|
on_pretrain_routine_start          	|  0.022465       	|1              	|  0.022465       	|  0.0043495      	|
on_batch_start                     	|  2.0735e-05     	|1000           	|  0.020735       	|  0.0040145      	|
on_before_zero_grad                	|  1.5172e-05     	|1000           	|  0.015172       	|  0.0029375      	|
on_batch_end                       	|  1.4166e-05     	|1000           	|  0.014166       	|  0.0027427      	|
on_before_optimizer_step           	|  1.1255e-05     	|1000           	|  0.011255       	|  0.0021792      	|
on_after_backward                  	|  1.1229e-05     	|1000           	|  0.011229       	|  0.002174       	|
on_before_backward                 	|  1.0963e-05     	|1000           	|  0.010963       	|  0.0021225      	|
configure_optimizers               	|  0.0082471      	|1              	|  0.0082471      	|  0.0015967      	|
training_step_end                  	|  6.0048e-06     	|1000           	|  0.0060048      	|  0.0011626      	|
on_validation_model_eval           	|  0.0045425      	|1              	|  0.0045425      	|  0.00087949     	|
on_validation_batch_end            	|  1.9605e-05     	|125            	|  0.0024506      	|  0.00047446     	|
on_epoch_start                     	|  1.0581e-05     	|168            	|  0.0017776      	|  0.00034417     	|
on_epoch_end                       	|  1.0289e-05     	|168            	|  0.0017285      	|  0.00033466     	|
on_validation_batch_start          	|  1.0506e-05     	|125            	|  0.0013133      	|  0.00025427     	|
on_train_start                     	|  0.00074311     	|1              	|  0.00074311     	|  0.00014387     	|
validation_step_end                	|  5.1238e-06     	|125            	|  0.00064047     	|  0.000124       	|
on_validation_start                	|  0.00033038     	|1              	|  0.00033038     	|  6.3965e-05     	|
on_validation_end                  	|  0.00011703     	|1              	|  0.00011703     	|  2.2658e-05     	|
on_before_accelerator_backend_setup	|  1.8389e-05     	|2              	|  3.6779e-05     	|  7.1208e-06     	|
on_configure_sharded_model         	|  1.5849e-05     	|2              	|  3.1699e-05     	|  6.1372e-06     	|
teardown                           	|  1.5133e-05     	|2              	|  3.0267e-05     	|  5.86e-06       	|
setup                              	|  1.4352e-05     	|2              	|  2.8703e-05     	|  5.5572e-06     	|
on_fit_end                         	|  2.2552e-05     	|1              	|  2.2552e-05     	|  4.3663e-06     	|
on_validation_epoch_end            	|  2.1711e-05     	|1              	|  2.1711e-05     	|  4.2035e-06     	|
on_fit_start                       	|  1.6431e-05     	|1              	|  1.6431e-05     	|  3.1812e-06     	|
prepare_data                       	|  5.996e-06      	|2              	|  1.1992e-05     	|  2.3218e-06     	|
on_val_dataloader                  	|  5.911e-06      	|2              	|  1.1822e-05     	|  2.2889e-06     	|
configure_callbacks                	|  5.5105e-06     	|2              	|  1.1021e-05     	|  2.1338e-06     	|
configure_sharded_model            	|  5.3e-06        	|2              	|  1.06e-05       	|  2.0523e-06     	|
on_pretrain_routine_end            	|  1.0119e-05     	|1              	|  1.0119e-05     	|  1.9591e-06     	|
on_validation_epoch_start          	|  9.908e-06      	|1              	|  9.908e-06      	|  1.9183e-06     	|
on_train_dataloader                	|  4.3481e-06     	|1              	|  4.3481e-06     	|  8.4184e-07     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1704.1         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.3336         	|167            	|  1558.7         	|  91.469         	|
run_training_batch                 	|  0.18895        	|8000           	|  1511.6         	|  88.706         	|
training_step_and_backward         	|  0.17873        	|8000           	|  1429.9         	|  83.907         	|
model_forward                      	|  0.094157       	|8000           	|  753.25         	|  44.203         	|
training_step                      	|  0.094022       	|8000           	|  752.17         	|  44.139         	|
backward                           	|  0.084386       	|8000           	|  675.08         	|  39.616         	|
optimizer_step_with_closure_0      	|  0.23776        	|1000           	|  237.76         	|  13.952         	|
run_validate_evaluation            	|  101.59         	|1              	|  101.59         	|  5.9614         	|
evaluation_step_and_end            	|  0.099581       	|1000           	|  99.581         	|  5.8436         	|
validation_step                    	|  0.099529       	|1000           	|  99.529         	|  5.8406         	|
get_train_batch                    	|  0.0033411      	|8167           	|  27.286         	|  1.6012         	|
fetch_next_train_batch             	|  0.0033234      	|8167           	|  27.142         	|  1.5928         	|
on_train_batch_end                 	|  0.0010753      	|8000           	|  8.602          	|  0.50479        	|
training_batch_to_device           	|  0.00021679     	|8000           	|  1.7343         	|  0.10177        	|
get_validate_batch                 	|  0.0014684      	|1001           	|  1.4699         	|  0.086258       	|
fetch_next_validate_batch          	|  0.0014546      	|1001           	|  1.4561         	|  0.085446       	|
zero_grad                          	|  0.0013721      	|1000           	|  1.3721         	|  0.08052        	|
on_train_batch_start               	|  9.0307e-05     	|8000           	|  0.72246        	|  0.042395       	|
evaluation_batch_to_device         	|  0.00020802     	|1000           	|  0.20802        	|  0.012207       	|
on_batch_start                     	|  1.5786e-05     	|8000           	|  0.12629        	|  0.0074108      	|
on_batch_end                       	|  1.3709e-05     	|8000           	|  0.10967        	|  0.0064356      	|
on_before_backward                 	|  1.334e-05      	|8000           	|  0.10672        	|  0.0062625      	|
on_after_backward                  	|  1.1923e-05     	|8000           	|  0.095385       	|  0.0055974      	|
on_train_epoch_start               	|  0.00028605     	|167            	|  0.04777        	|  0.0028032      	|
training_step_end                  	|  5.8314e-06     	|8000           	|  0.046651       	|  0.0027376      	|
on_train_epoch_end                 	|  0.00026759     	|167            	|  0.044687       	|  0.0026223      	|
on_train_end                       	|  0.025865       	|1              	|  0.025865       	|  0.0015178      	|
on_pretrain_routine_start          	|  0.022435       	|1              	|  0.022435       	|  0.0013165      	|
on_validation_batch_end            	|  1.9087e-05     	|1000           	|  0.019087       	|  0.0011201      	|
on_before_zero_grad                	|  1.6514e-05     	|1000           	|  0.016514       	|  0.00096906     	|
on_before_optimizer_step           	|  1.3725e-05     	|1000           	|  0.013725       	|  0.00080539     	|
on_validation_batch_start          	|  1.0402e-05     	|1000           	|  0.010402       	|  0.00061039     	|
configure_optimizers               	|  0.0081189      	|1              	|  0.0081189      	|  0.00047643     	|
validation_step_end                	|  4.6887e-06     	|1000           	|  0.0046887      	|  0.00027514     	|
on_validation_model_eval           	|  0.0040755      	|1              	|  0.0040755      	|  0.00023916     	|
on_epoch_start                     	|  1.1002e-05     	|168            	|  0.0018483      	|  0.00010846     	|
on_epoch_end                       	|  1.0771e-05     	|168            	|  0.0018095      	|  0.00010619     	|
on_train_start                     	|  0.00074223     	|1              	|  0.00074223     	|  4.3556e-05     	|
on_validation_start                	|  0.00032184     	|1              	|  0.00032184     	|  1.8886e-05     	|
on_validation_end                  	|  0.00011124     	|1              	|  0.00011124     	|  6.5277e-06     	|
on_before_accelerator_backend_setup	|  2.0178e-05     	|2              	|  4.0355e-05     	|  2.3681e-06     	|
teardown                           	|  1.5815e-05     	|2              	|  3.163e-05      	|  1.8561e-06     	|
setup                              	|  1.4803e-05     	|2              	|  2.9605e-05     	|  1.7373e-06     	|
on_configure_sharded_model         	|  1.4301e-05     	|2              	|  2.8603e-05     	|  1.6785e-06     	|
on_fit_end                         	|  2.2452e-05     	|1              	|  2.2452e-05     	|  1.3175e-06     	|
on_fit_start                       	|  1.8575e-05     	|1              	|  1.8575e-05     	|  1.09e-06       	|
on_validation_epoch_end            	|  1.7533e-05     	|1              	|  1.7533e-05     	|  1.0289e-06     	|
configure_callbacks                	|  7.7795e-06     	|2              	|  1.5559e-05     	|  9.1303e-07     	|
on_pretrain_routine_end            	|  1.3365e-05     	|1              	|  1.3365e-05     	|  7.8429e-07     	|
prepare_data                       	|  6.4269e-06     	|2              	|  1.2854e-05     	|  7.543e-07      	|
on_val_dataloader                  	|  5.996e-06      	|2              	|  1.1992e-05     	|  7.0371e-07     	|
on_validation_epoch_start          	|  9.9679e-06     	|1              	|  9.9679e-06     	|  5.8494e-07     	|
configure_sharded_model            	|  4.824e-06      	|2              	|  9.648e-06      	|  5.6617e-07     	|
on_train_dataloader                	|  4.0971e-06     	|1              	|  4.0971e-06     	|  2.4043e-07     	|
