VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  507.24         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.5716         	|167            	|  429.45         	|  84.664         	|
run_training_batch                 	|  0.37587        	|1000           	|  375.87         	|  74.101         	|
optimizer_step_with_closure_0      	|  0.37313        	|1000           	|  373.13         	|  73.561         	|
training_step_and_backward         	|  0.29413        	|1000           	|  294.13         	|  57.986         	|
model_forward                      	|  0.17059        	|1000           	|  170.59         	|  33.631         	|
training_step                      	|  0.17044        	|1000           	|  170.44         	|  33.601         	|
backward                           	|  0.12219        	|1000           	|  122.19         	|  24.089         	|
run_validate_evaluation            	|  34.814         	|1              	|  34.814         	|  6.8634         	|
evaluation_step_and_end            	|  0.27365        	|125            	|  34.206         	|  6.7436         	|
validation_step                    	|  0.2736         	|125            	|  34.199         	|  6.7422         	|
get_train_batch                    	|  0.014131       	|1167           	|  16.491         	|  3.2512         	|
fetch_next_train_batch             	|  0.014107       	|1167           	|  16.463         	|  3.2457         	|
zero_grad                          	|  0.0013221      	|1000           	|  1.3221         	|  0.26065        	|
on_train_batch_end                 	|  0.0012363      	|1000           	|  1.2363         	|  0.24373        	|
get_validate_batch                 	|  0.0025629      	|126            	|  0.32293        	|  0.063664       	|
fetch_next_validate_batch          	|  0.0025485      	|126            	|  0.32111        	|  0.063305       	|
training_batch_to_device           	|  0.00031446     	|1000           	|  0.31446        	|  0.061994       	|
on_train_batch_start               	|  9.513e-05      	|1000           	|  0.09513        	|  0.018754       	|
on_train_epoch_start               	|  0.00027926     	|167            	|  0.046636       	|  0.0091941      	|
on_train_epoch_end                 	|  0.00026533     	|167            	|  0.04431        	|  0.0087355      	|
evaluation_batch_to_device         	|  0.00023308     	|125            	|  0.029136       	|  0.0057439      	|
on_train_end                       	|  0.024651       	|1              	|  0.024651       	|  0.0048598      	|
on_pretrain_routine_start          	|  0.022596       	|1              	|  0.022596       	|  0.0044547      	|
on_batch_start                     	|  2.1059e-05     	|1000           	|  0.021059       	|  0.0041516      	|
on_before_zero_grad                	|  1.5399e-05     	|1000           	|  0.015399       	|  0.0030358      	|
on_batch_end                       	|  1.3804e-05     	|1000           	|  0.013804       	|  0.0027213      	|
on_after_backward                  	|  1.1807e-05     	|1000           	|  0.011807       	|  0.0023277      	|
on_before_optimizer_step           	|  1.1354e-05     	|1000           	|  0.011354       	|  0.0022383      	|
on_before_backward                 	|  1.072e-05      	|1000           	|  0.01072        	|  0.0021133      	|
configure_optimizers               	|  0.0085719      	|1              	|  0.0085719      	|  0.0016899      	|
training_step_end                  	|  6.9868e-06     	|1000           	|  0.0069868      	|  0.0013774      	|
on_validation_model_eval           	|  0.0040011      	|1              	|  0.0040011      	|  0.00078879     	|
on_validation_batch_end            	|  2.1192e-05     	|125            	|  0.002649       	|  0.00052224     	|
on_epoch_end                       	|  1.0532e-05     	|168            	|  0.0017693      	|  0.00034882     	|
on_epoch_start                     	|  1.0396e-05     	|168            	|  0.0017465      	|  0.00034432     	|
on_validation_batch_start          	|  1.0418e-05     	|125            	|  0.0013023      	|  0.00025674     	|
on_train_start                     	|  0.00075695     	|1              	|  0.00075695     	|  0.00014923     	|
validation_step_end                	|  5.1657e-06     	|125            	|  0.00064571     	|  0.0001273      	|
on_validation_start                	|  0.00032586     	|1              	|  0.00032586     	|  6.4241e-05     	|
on_validation_end                  	|  0.00011005     	|1              	|  0.00011005     	|  2.1695e-05     	|
on_before_accelerator_backend_setup	|  2.1696e-05     	|2              	|  4.3391e-05     	|  8.5543e-06     	|
teardown                           	|  1.5203e-05     	|2              	|  3.0406e-05     	|  5.9944e-06     	|
on_configure_sharded_model         	|  1.4542e-05     	|2              	|  2.9084e-05     	|  5.7338e-06     	|
setup                              	|  1.4186e-05     	|2              	|  2.8373e-05     	|  5.5936e-06     	|
on_fit_end                         	|  2.3484e-05     	|1              	|  2.3484e-05     	|  4.6297e-06     	|
on_validation_epoch_end            	|  2.0578e-05     	|1              	|  2.0578e-05     	|  4.0568e-06     	|
on_fit_start                       	|  1.6651e-05     	|1              	|  1.6651e-05     	|  3.2827e-06     	|
prepare_data                       	|  6.883e-06      	|2              	|  1.3766e-05     	|  2.7139e-06     	|
on_val_dataloader                  	|  5.796e-06      	|2              	|  1.1592e-05     	|  2.2853e-06     	|
on_pretrain_routine_end            	|  1.082e-05      	|1              	|  1.082e-05      	|  2.1331e-06     	|
configure_sharded_model            	|  5.265e-06      	|2              	|  1.053e-05      	|  2.0759e-06     	|
configure_callbacks                	|  4.9845e-06     	|2              	|  9.969e-06      	|  1.9653e-06     	|
on_validation_epoch_start          	|  9.748e-06      	|1              	|  9.748e-06      	|  1.9218e-06     	|
on_train_dataloader                	|  4.569e-06      	|1              	|  4.569e-06      	|  9.0074e-07     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1679.2         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.1836         	|167            	|  1533.7         	|  91.334         	|
run_training_batch                 	|  0.18581        	|8000           	|  1486.5         	|  88.524         	|
training_step_and_backward         	|  0.17573        	|8000           	|  1405.8         	|  83.722         	|
model_forward                      	|  0.094059       	|8000           	|  752.47         	|  44.812         	|
training_step                      	|  0.093926       	|8000           	|  751.41         	|  44.748         	|
backward                           	|  0.081489       	|8000           	|  651.91         	|  38.823         	|
optimizer_step_with_closure_0      	|  0.23367        	|1000           	|  233.67         	|  13.916         	|
run_validate_evaluation            	|  101.67         	|1              	|  101.67         	|  6.0547         	|
evaluation_step_and_end            	|  0.099658       	|1000           	|  99.658         	|  5.9349         	|
validation_step                    	|  0.099607       	|1000           	|  99.607         	|  5.9319         	|
get_train_batch                    	|  0.0033553      	|8167           	|  27.403         	|  1.6319         	|
fetch_next_train_batch             	|  0.0033381      	|8167           	|  27.263         	|  1.6236         	|
on_train_batch_end                 	|  0.0010768      	|8000           	|  8.6143         	|  0.51301        	|
training_batch_to_device           	|  0.00021962     	|8000           	|  1.757          	|  0.10463        	|
get_validate_batch                 	|  0.0014717      	|1001           	|  1.4732         	|  0.087731       	|
fetch_next_validate_batch          	|  0.0014583      	|1001           	|  1.4597         	|  0.086931       	|
zero_grad                          	|  0.0013094      	|1000           	|  1.3094         	|  0.077976       	|
on_train_batch_start               	|  9.0942e-05     	|8000           	|  0.72753        	|  0.043327       	|
evaluation_batch_to_device         	|  0.00021045     	|1000           	|  0.21045        	|  0.012533       	|
on_batch_start                     	|  1.5402e-05     	|8000           	|  0.12321        	|  0.0073376      	|
on_batch_end                       	|  1.2865e-05     	|8000           	|  0.10292        	|  0.0061291      	|
on_before_backward                 	|  1.2739e-05     	|8000           	|  0.10192        	|  0.0060693      	|
on_after_backward                  	|  1.1335e-05     	|8000           	|  0.090676       	|  0.0054         	|
on_train_epoch_start               	|  0.00029052     	|167            	|  0.048517       	|  0.0028893      	|
training_step_end                  	|  5.7425e-06     	|8000           	|  0.04594        	|  0.0027358      	|
on_train_epoch_end                 	|  0.00025643     	|167            	|  0.042824       	|  0.0025503      	|
on_train_end                       	|  0.026792       	|1              	|  0.026792       	|  0.0015955      	|
on_pretrain_routine_start          	|  0.022945       	|1              	|  0.022945       	|  0.0013665      	|
on_validation_batch_end            	|  1.7877e-05     	|1000           	|  0.017877       	|  0.0010646      	|
on_before_zero_grad                	|  1.5749e-05     	|1000           	|  0.015749       	|  0.00093789     	|
on_before_optimizer_step           	|  1.2436e-05     	|1000           	|  0.012436       	|  0.00074057     	|
on_validation_batch_start          	|  9.9829e-06     	|1000           	|  0.0099829      	|  0.00059451     	|
configure_optimizers               	|  0.0084384      	|1              	|  0.0084384      	|  0.00050253     	|
on_validation_model_eval           	|  0.0044709      	|1              	|  0.0044709      	|  0.00026625     	|
validation_step_end                	|  4.447e-06      	|1000           	|  0.004447       	|  0.00026483     	|
on_epoch_end                       	|  1.1093e-05     	|168            	|  0.0018636      	|  0.00011098     	|
on_epoch_start                     	|  1.0554e-05     	|168            	|  0.0017731      	|  0.0001056      	|
on_train_start                     	|  0.00075286     	|1              	|  0.00075286     	|  4.4835e-05     	|
on_validation_start                	|  0.00032625     	|1              	|  0.00032625     	|  1.9429e-05     	|
on_validation_end                  	|  0.00010641     	|1              	|  0.00010641     	|  6.337e-06      	|
on_before_accelerator_backend_setup	|  1.8961e-05     	|2              	|  3.7921e-05     	|  2.2583e-06     	|
teardown                           	|  1.5804e-05     	|2              	|  3.1609e-05     	|  1.8824e-06     	|
setup                              	|  1.3686e-05     	|2              	|  2.7371e-05     	|  1.63e-06       	|
on_configure_sharded_model         	|  1.3349e-05     	|2              	|  2.6699e-05     	|  1.59e-06       	|
on_fit_end                         	|  2.5157e-05     	|1              	|  2.5157e-05     	|  1.4982e-06     	|
on_validation_epoch_end            	|  1.9006e-05     	|1              	|  1.9006e-05     	|  1.1319e-06     	|
on_fit_start                       	|  1.7833e-05     	|1              	|  1.7833e-05     	|  1.062e-06      	|
on_pretrain_routine_end            	|  1.1792e-05     	|1              	|  1.1792e-05     	|  7.0224e-07     	|
on_val_dataloader                  	|  5.6505e-06     	|2              	|  1.1301e-05     	|  6.73e-07       	|
prepare_data                       	|  5.6406e-06     	|2              	|  1.1281e-05     	|  6.7182e-07     	|
configure_callbacks                	|  5.1045e-06     	|2              	|  1.0209e-05     	|  6.0797e-07     	|
configure_sharded_model            	|  4.924e-06      	|2              	|  9.848e-06      	|  5.8648e-07     	|
on_validation_epoch_start          	|  9.5281e-06     	|1              	|  9.5281e-06     	|  5.6743e-07     	|
on_train_dataloader                	|  4.238e-06      	|1              	|  4.238e-06      	|  2.5238e-07     	|
