VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  511.38         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.5921         	|167            	|  432.88         	|  84.65          	|
run_training_batch                 	|  0.37925        	|1000           	|  379.25         	|  74.163         	|
optimizer_step_with_closure_0      	|  0.37656        	|1000           	|  376.56         	|  73.636         	|
training_step_and_backward         	|  0.29712        	|1000           	|  297.12         	|  58.102         	|
model_forward                      	|  0.1721         	|1000           	|  172.1          	|  33.654         	|
training_step                      	|  0.17195        	|1000           	|  171.95         	|  33.625         	|
backward                           	|  0.1237         	|1000           	|  123.7          	|  24.19          	|
run_validate_evaluation            	|  35.219         	|1              	|  35.219         	|  6.8871         	|
evaluation_step_and_end            	|  0.27702        	|125            	|  34.628         	|  6.7715         	|
validation_step                    	|  0.27697        	|125            	|  34.621         	|  6.7701         	|
get_train_batch                    	|  0.014151       	|1167           	|  16.514         	|  3.2294         	|
fetch_next_train_batch             	|  0.014127       	|1167           	|  16.486         	|  3.2239         	|
zero_grad                          	|  0.0012904      	|1000           	|  1.2904         	|  0.25233        	|
on_train_batch_end                 	|  0.0012333      	|1000           	|  1.2333         	|  0.24117        	|
training_batch_to_device           	|  0.00031102     	|1000           	|  0.31102        	|  0.06082        	|
get_validate_batch                 	|  0.0024454      	|126            	|  0.30812        	|  0.060253       	|
fetch_next_validate_batch          	|  0.0024311      	|126            	|  0.30632        	|  0.059901       	|
on_train_batch_start               	|  9.6086e-05     	|1000           	|  0.096086       	|  0.01879        	|
on_train_epoch_start               	|  0.00027686     	|167            	|  0.046235       	|  0.0090413      	|
on_train_epoch_end                 	|  0.00026861     	|167            	|  0.044858       	|  0.0087719      	|
evaluation_batch_to_device         	|  0.00021926     	|125            	|  0.027408       	|  0.0053596      	|
on_train_end                       	|  0.024833       	|1              	|  0.024833       	|  0.0048561      	|
on_pretrain_routine_start          	|  0.022241       	|1              	|  0.022241       	|  0.0043492      	|
on_batch_start                     	|  2.1075e-05     	|1000           	|  0.021075       	|  0.0041213      	|
on_before_zero_grad                	|  1.4966e-05     	|1000           	|  0.014966       	|  0.0029266      	|
on_batch_end                       	|  1.4189e-05     	|1000           	|  0.014189       	|  0.0027746      	|
on_after_backward                  	|  1.1191e-05     	|1000           	|  0.011191       	|  0.0021884      	|
on_before_optimizer_step           	|  1.077e-05      	|1000           	|  0.01077        	|  0.0021061      	|
on_before_backward                 	|  1.0595e-05     	|1000           	|  0.010595       	|  0.0020718      	|
configure_optimizers               	|  0.0080945      	|1              	|  0.0080945      	|  0.0015829      	|
training_step_end                  	|  6.7079e-06     	|1000           	|  0.0067079      	|  0.0013117      	|
on_validation_model_eval           	|  0.0040733      	|1              	|  0.0040733      	|  0.00079653     	|
on_validation_batch_end            	|  2.1391e-05     	|125            	|  0.0026738      	|  0.00052287     	|
on_epoch_end                       	|  1.1015e-05     	|168            	|  0.0018506      	|  0.00036188     	|
on_epoch_start                     	|  1.0678e-05     	|168            	|  0.0017939      	|  0.00035079     	|
on_validation_batch_start          	|  1.0513e-05     	|125            	|  0.0013142      	|  0.00025698     	|
on_train_start                     	|  0.00072556     	|1              	|  0.00072556     	|  0.00014188     	|
validation_step_end                	|  5.4405e-06     	|125            	|  0.00068006     	|  0.00013299     	|
on_validation_start                	|  0.0003277      	|1              	|  0.0003277      	|  6.4083e-05     	|
on_validation_end                  	|  0.00010431     	|1              	|  0.00010431     	|  2.0399e-05     	|
on_before_accelerator_backend_setup	|  1.8435e-05     	|2              	|  3.6869e-05     	|  7.2098e-06     	|
teardown                           	|  1.4903e-05     	|2              	|  2.9806e-05     	|  5.8286e-06     	|
setup                              	|  1.4056e-05     	|2              	|  2.8113e-05     	|  5.4975e-06     	|
on_configure_sharded_model         	|  1.3736e-05     	|2              	|  2.7472e-05     	|  5.3722e-06     	|
on_fit_end                         	|  2.7101e-05     	|1              	|  2.7101e-05     	|  5.2996e-06     	|
on_validation_epoch_end            	|  1.8996e-05     	|1              	|  1.8996e-05     	|  3.7147e-06     	|
on_fit_start                       	|  1.628e-05      	|1              	|  1.628e-05      	|  3.1836e-06     	|
prepare_data                       	|  6.201e-06      	|2              	|  1.2402e-05     	|  2.4252e-06     	|
on_val_dataloader                  	|  6.0965e-06     	|2              	|  1.2193e-05     	|  2.3843e-06     	|
configure_callbacks                	|  5.9159e-06     	|2              	|  1.1832e-05     	|  2.3137e-06     	|
on_pretrain_routine_end            	|  9.899e-06      	|1              	|  9.899e-06      	|  1.9358e-06     	|
on_validation_epoch_start          	|  9.177e-06      	|1              	|  9.177e-06      	|  1.7946e-06     	|
configure_sharded_model            	|  4.3581e-06     	|2              	|  8.7161e-06     	|  1.7044e-06     	|
on_train_dataloader                	|  4.3791e-06     	|1              	|  4.3791e-06     	|  8.5633e-07     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1695.0         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.2822         	|167            	|  1550.1         	|  91.453         	|
run_training_batch                 	|  0.18791        	|8000           	|  1503.3         	|  88.689         	|
training_step_and_backward         	|  0.17772        	|8000           	|  1421.8         	|  83.881         	|
model_forward                      	|  0.094105       	|8000           	|  752.84         	|  44.415         	|
training_step                      	|  0.09397        	|8000           	|  751.76         	|  44.352         	|
backward                           	|  0.083432       	|8000           	|  667.45         	|  39.378         	|
optimizer_step_with_closure_0      	|  0.23625        	|1000           	|  236.25         	|  13.938         	|
run_validate_evaluation            	|  101.62         	|1              	|  101.62         	|  5.9953         	|
evaluation_step_and_end            	|  0.099656       	|1000           	|  99.656         	|  5.8794         	|
validation_step                    	|  0.099606       	|1000           	|  99.606         	|  5.8764         	|
get_train_batch                    	|  0.0033302      	|8167           	|  27.198         	|  1.6046         	|
fetch_next_train_batch             	|  0.0033127      	|8167           	|  27.055         	|  1.5961         	|
on_train_batch_end                 	|  0.001082       	|8000           	|  8.6559         	|  0.51067        	|
training_batch_to_device           	|  0.00021816     	|8000           	|  1.7453         	|  0.10297        	|
get_validate_batch                 	|  0.0014352      	|1001           	|  1.4367         	|  0.084759       	|
fetch_next_validate_batch          	|  0.0014214      	|1001           	|  1.4228         	|  0.083943       	|
zero_grad                          	|  0.0013352      	|1000           	|  1.3352         	|  0.078772       	|
on_train_batch_start               	|  8.9803e-05     	|8000           	|  0.71842        	|  0.042385       	|
evaluation_batch_to_device         	|  0.00020471     	|1000           	|  0.20471        	|  0.012077       	|
on_batch_start                     	|  1.5038e-05     	|8000           	|  0.1203         	|  0.0070976      	|
on_batch_end                       	|  1.3342e-05     	|8000           	|  0.10674        	|  0.0062972      	|
on_before_backward                 	|  1.266e-05      	|8000           	|  0.10128        	|  0.0059755      	|
on_after_backward                  	|  1.1234e-05     	|8000           	|  0.08987        	|  0.0053021      	|
on_train_epoch_start               	|  0.00029176     	|167            	|  0.048724       	|  0.0028746      	|
training_step_end                  	|  5.887e-06      	|8000           	|  0.047096       	|  0.0027785      	|
on_train_epoch_end                 	|  0.00026155     	|167            	|  0.043678       	|  0.0025769      	|
on_train_end                       	|  0.025969       	|1              	|  0.025969       	|  0.0015321      	|
on_pretrain_routine_start          	|  0.022674       	|1              	|  0.022674       	|  0.0013377      	|
on_validation_batch_end            	|  1.7674e-05     	|1000           	|  0.017674       	|  0.0010427      	|
on_before_zero_grad                	|  1.5428e-05     	|1000           	|  0.015428       	|  0.00091022     	|
on_before_optimizer_step           	|  1.268e-05      	|1000           	|  0.01268        	|  0.00074808     	|
on_validation_batch_start          	|  9.5361e-06     	|1000           	|  0.0095361      	|  0.0005626      	|
configure_optimizers               	|  0.0088318      	|1              	|  0.0088318      	|  0.00052105     	|
validation_step_end                	|  4.6761e-06     	|1000           	|  0.0046761      	|  0.00027588     	|
on_validation_model_eval           	|  0.0040052      	|1              	|  0.0040052      	|  0.00023629     	|
on_epoch_start                     	|  1.0697e-05     	|168            	|  0.001797       	|  0.00010602     	|
on_epoch_end                       	|  1.0428e-05     	|168            	|  0.0017519      	|  0.00010336     	|
on_train_start                     	|  0.00046952     	|1              	|  0.00046952     	|  2.77e-05       	|
on_validation_start                	|  0.00032186     	|1              	|  0.00032186     	|  1.8989e-05     	|
on_validation_end                  	|  0.00010842     	|1              	|  0.00010842     	|  6.3966e-06     	|
on_before_accelerator_backend_setup	|  1.9041e-05     	|2              	|  3.8082e-05     	|  2.2467e-06     	|
teardown                           	|  1.5479e-05     	|2              	|  3.0958e-05     	|  1.8264e-06     	|
on_configure_sharded_model         	|  1.3861e-05     	|2              	|  2.7722e-05     	|  1.6355e-06     	|
setup                              	|  1.3796e-05     	|2              	|  2.7592e-05     	|  1.6278e-06     	|
on_fit_end                         	|  2.125e-05      	|1              	|  2.125e-05      	|  1.2537e-06     	|
on_validation_epoch_end            	|  2.096e-05      	|1              	|  2.096e-05      	|  1.2366e-06     	|
on_fit_start                       	|  1.7843e-05     	|1              	|  1.7843e-05     	|  1.0527e-06     	|
on_pretrain_routine_end            	|  1.3054e-05     	|1              	|  1.3054e-05     	|  7.7014e-07     	|
prepare_data                       	|  6.4465e-06     	|2              	|  1.2893e-05     	|  7.6065e-07     	|
on_val_dataloader                  	|  6.2265e-06     	|2              	|  1.2453e-05     	|  7.3469e-07     	|
configure_callbacks                	|  5.6605e-06     	|2              	|  1.1321e-05     	|  6.679e-07      	|
configure_sharded_model            	|  5.0996e-06     	|2              	|  1.0199e-05     	|  6.0172e-07     	|
on_validation_epoch_start          	|  9.5279e-06     	|1              	|  9.5279e-06     	|  5.6212e-07     	|
on_train_dataloader                	|  4.549e-06      	|1              	|  4.549e-06      	|  2.6838e-07     	|
