VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  507.54         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.5677         	|167            	|  428.81         	|  84.488         	|
run_training_batch                 	|  0.37528        	|1000           	|  375.28         	|  73.941         	|
optimizer_step_with_closure_0      	|  0.37258        	|1000           	|  372.58         	|  73.408         	|
training_step_and_backward         	|  0.29332        	|1000           	|  293.32         	|  57.793         	|
model_forward                      	|  0.1703         	|1000           	|  170.3          	|  33.554         	|
training_step                      	|  0.17016        	|1000           	|  170.16         	|  33.526         	|
backward                           	|  0.12169        	|1000           	|  121.69         	|  23.977         	|
run_validate_evaluation            	|  35.063         	|1              	|  35.063         	|  6.9084         	|
evaluation_step_and_end            	|  0.27559        	|125            	|  34.449         	|  6.7874         	|
validation_step                    	|  0.27554        	|125            	|  34.442         	|  6.7861         	|
get_train_batch                    	|  0.014021       	|1167           	|  16.363         	|  3.224          	|
fetch_next_train_batch             	|  0.013998       	|1167           	|  16.336         	|  3.2186         	|
zero_grad                          	|  0.0013038      	|1000           	|  1.3038         	|  0.25689        	|
on_train_batch_end                 	|  0.001216       	|1000           	|  1.216          	|  0.23958        	|
get_validate_batch                 	|  0.0026947      	|126            	|  0.33953        	|  0.066898       	|
fetch_next_validate_batch          	|  0.0026799      	|126            	|  0.33767        	|  0.066531       	|
training_batch_to_device           	|  0.00031155     	|1000           	|  0.31155        	|  0.061385       	|
on_train_batch_start               	|  9.4851e-05     	|1000           	|  0.094851       	|  0.018688       	|
on_train_epoch_start               	|  0.00027044     	|167            	|  0.045164       	|  0.0088986      	|
on_train_epoch_end                 	|  0.000254       	|167            	|  0.042418       	|  0.0083576      	|
evaluation_batch_to_device         	|  0.00021683     	|125            	|  0.027104       	|  0.0053403      	|
on_train_end                       	|  0.024847       	|1              	|  0.024847       	|  0.0048955      	|
on_pretrain_routine_start          	|  0.022717       	|1              	|  0.022717       	|  0.0044759      	|
on_batch_start                     	|  2.0702e-05     	|1000           	|  0.020702       	|  0.004079       	|
on_before_zero_grad                	|  1.5533e-05     	|1000           	|  0.015533       	|  0.0030605      	|
on_batch_end                       	|  1.4068e-05     	|1000           	|  0.014068       	|  0.0027719      	|
on_after_backward                  	|  1.174e-05      	|1000           	|  0.01174        	|  0.0023132      	|
on_before_optimizer_step           	|  1.122e-05      	|1000           	|  0.01122        	|  0.0022106      	|
on_before_backward                 	|  1.0968e-05     	|1000           	|  0.010968       	|  0.002161       	|
configure_optimizers               	|  0.0086599      	|1              	|  0.0086599      	|  0.0017063      	|
training_step_end                  	|  6.3952e-06     	|1000           	|  0.0063952      	|  0.00126        	|
on_validation_model_eval           	|  0.0045777      	|1              	|  0.0045777      	|  0.00090194     	|
on_validation_batch_end            	|  2.2568e-05     	|125            	|  0.002821       	|  0.00055582     	|
on_epoch_start                     	|  1.0763e-05     	|168            	|  0.0018081      	|  0.00035625     	|
on_epoch_end                       	|  1.0319e-05     	|168            	|  0.0017337      	|  0.00034158     	|
on_validation_batch_start          	|  1.0335e-05     	|125            	|  0.0012918      	|  0.00025453     	|
on_train_start                     	|  0.00075408     	|1              	|  0.00075408     	|  0.00014858     	|
validation_step_end                	|  5.0846e-06     	|125            	|  0.00063558     	|  0.00012523     	|
on_validation_start                	|  0.00033211     	|1              	|  0.00033211     	|  6.5436e-05     	|
on_validation_end                  	|  0.00010604     	|1              	|  0.00010604     	|  2.0893e-05     	|
on_before_accelerator_backend_setup	|  1.8715e-05     	|2              	|  3.743e-05      	|  7.3748e-06     	|
teardown                           	|  1.5444e-05     	|2              	|  3.0887e-05     	|  6.0857e-06     	|
setup                              	|  1.4963e-05     	|2              	|  2.9926e-05     	|  5.8963e-06     	|
on_configure_sharded_model         	|  1.4372e-05     	|2              	|  2.8744e-05     	|  5.6634e-06     	|
on_fit_end                         	|  2.139e-05      	|1              	|  2.139e-05      	|  4.2145e-06     	|
on_fit_start                       	|  1.7874e-05     	|1              	|  1.7874e-05     	|  3.5217e-06     	|
on_validation_epoch_end            	|  1.7492e-05     	|1              	|  1.7492e-05     	|  3.4464e-06     	|
on_val_dataloader                  	|  6.1165e-06     	|2              	|  1.2233e-05     	|  2.4102e-06     	|
prepare_data                       	|  6.071e-06      	|2              	|  1.2142e-05     	|  2.3923e-06     	|
configure_callbacks                	|  5.23e-06       	|2              	|  1.046e-05      	|  2.0609e-06     	|
configure_sharded_model            	|  5.14e-06       	|2              	|  1.028e-05      	|  2.0255e-06     	|
on_pretrain_routine_end            	|  9.878e-06      	|1              	|  9.878e-06      	|  1.9462e-06     	|
on_validation_epoch_start          	|  9.5479e-06     	|1              	|  9.5479e-06     	|  1.8812e-06     	|
on_train_dataloader                	|  4.799e-06      	|1              	|  4.799e-06      	|  9.4554e-07     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1685.4         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.2234         	|167            	|  1540.3         	|  91.389         	|
run_training_batch                 	|  0.18665        	|8000           	|  1493.2         	|  88.594         	|
training_step_and_backward         	|  0.17663        	|8000           	|  1413.0         	|  83.838         	|
model_forward                      	|  0.094002       	|8000           	|  752.02         	|  44.619         	|
training_step                      	|  0.093868       	|8000           	|  750.94         	|  44.555         	|
backward                           	|  0.082445       	|8000           	|  659.56         	|  39.133         	|
optimizer_step_with_closure_0      	|  0.23407        	|1000           	|  234.07         	|  13.888         	|
run_validate_evaluation            	|  101.57         	|1              	|  101.57         	|  6.0265         	|
evaluation_step_and_end            	|  0.099601       	|1000           	|  99.601         	|  5.9095         	|
validation_step                    	|  0.099552       	|1000           	|  99.552         	|  5.9066         	|
get_train_batch                    	|  0.0033462      	|8167           	|  27.329         	|  1.6215         	|
fetch_next_train_batch             	|  0.0033288      	|8167           	|  27.186         	|  1.613          	|
on_train_batch_end                 	|  0.0010824      	|8000           	|  8.6592         	|  0.51377        	|
training_batch_to_device           	|  0.00021558     	|8000           	|  1.7247         	|  0.10233        	|
get_validate_batch                 	|  0.0014383      	|1001           	|  1.4397         	|  0.085422       	|
fetch_next_validate_batch          	|  0.001425       	|1001           	|  1.4264         	|  0.08463        	|
zero_grad                          	|  0.0013148      	|1000           	|  1.3148         	|  0.078007       	|
on_train_batch_start               	|  9.0251e-05     	|8000           	|  0.722          	|  0.042838       	|
evaluation_batch_to_device         	|  0.00020765     	|1000           	|  0.20765        	|  0.01232        	|
on_batch_start                     	|  1.4857e-05     	|8000           	|  0.11886        	|  0.007052       	|
on_before_backward                 	|  1.3491e-05     	|8000           	|  0.10793        	|  0.0064037      	|
on_batch_end                       	|  1.3187e-05     	|8000           	|  0.1055         	|  0.0062594      	|
on_after_backward                  	|  1.1509e-05     	|8000           	|  0.092071       	|  0.0054627      	|
training_step_end                  	|  6.1909e-06     	|8000           	|  0.049527       	|  0.0029385      	|
on_train_epoch_start               	|  0.00028986     	|167            	|  0.048406       	|  0.002872       	|
on_train_epoch_end                 	|  0.00026848     	|167            	|  0.044836       	|  0.0026602      	|
on_train_end                       	|  0.025255       	|1              	|  0.025255       	|  0.0014985      	|
on_pretrain_routine_start          	|  0.022949       	|1              	|  0.022949       	|  0.0013616      	|
on_validation_batch_end            	|  1.8446e-05     	|1000           	|  0.018446       	|  0.0010944      	|
on_before_zero_grad                	|  1.6457e-05     	|1000           	|  0.016457       	|  0.00097642     	|
on_before_optimizer_step           	|  1.3473e-05     	|1000           	|  0.013473       	|  0.00079936     	|
on_validation_batch_start          	|  9.7906e-06     	|1000           	|  0.0097906      	|  0.00058089     	|
configure_optimizers               	|  0.0089691      	|1              	|  0.0089691      	|  0.00053215     	|
validation_step_end                	|  4.8106e-06     	|1000           	|  0.0048106      	|  0.00028542     	|
on_validation_model_eval           	|  0.0039593      	|1              	|  0.0039593      	|  0.00023491     	|
on_epoch_end                       	|  1.3194e-05     	|168            	|  0.0022166      	|  0.00013151     	|
on_epoch_start                     	|  1.1183e-05     	|168            	|  0.0018788      	|  0.00011147     	|
on_train_start                     	|  0.00076157     	|1              	|  0.00076157     	|  4.5185e-05     	|
on_validation_start                	|  0.00031495     	|1              	|  0.00031495     	|  1.8687e-05     	|
on_validation_end                  	|  0.0001127      	|1              	|  0.0001127      	|  6.6867e-06     	|
on_before_accelerator_backend_setup	|  2.0248e-05     	|2              	|  4.0495e-05     	|  2.4027e-06     	|
teardown                           	|  1.4983e-05     	|2              	|  2.9966e-05     	|  1.7779e-06     	|
setup                              	|  1.4633e-05     	|2              	|  2.9265e-05     	|  1.7364e-06     	|
on_configure_sharded_model         	|  1.341e-05      	|2              	|  2.682e-05      	|  1.5913e-06     	|
on_fit_end                         	|  2.2893e-05     	|1              	|  2.2893e-05     	|  1.3583e-06     	|
on_fit_start                       	|  1.8975e-05     	|1              	|  1.8975e-05     	|  1.1258e-06     	|
on_validation_epoch_end            	|  1.7362e-05     	|1              	|  1.7362e-05     	|  1.0301e-06     	|
prepare_data                       	|  6.733e-06      	|2              	|  1.3466e-05     	|  7.9896e-07     	|
on_pretrain_routine_end            	|  1.3075e-05     	|1              	|  1.3075e-05     	|  7.7577e-07     	|
on_val_dataloader                  	|  6.017e-06      	|2              	|  1.2034e-05     	|  7.1401e-07     	|
configure_callbacks                	|  5.9861e-06     	|2              	|  1.1972e-05     	|  7.1033e-07     	|
configure_sharded_model            	|  5.3202e-06     	|2              	|  1.064e-05      	|  6.3131e-07     	|
on_validation_epoch_start          	|  9.4171e-06     	|1              	|  9.4171e-06     	|  5.5873e-07     	|
on_train_dataloader                	|  4.1469e-06     	|1              	|  4.1469e-06     	|  2.4605e-07     	|
