VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  420.22         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.7575         	|63             	|  362.72         	|  86.318         	|
run_training_batch                 	|  0.33602        	|1000           	|  336.02         	|  79.963         	|
optimizer_step_with_closure_0      	|  0.33333        	|1000           	|  333.33         	|  79.323         	|
training_step_and_backward         	|  0.24869        	|1000           	|  248.69         	|  59.181         	|
model_forward                      	|  0.15433        	|1000           	|  154.33         	|  36.727         	|
training_step                      	|  0.15419        	|1000           	|  154.19         	|  36.694         	|
backward                           	|  0.093018       	|1000           	|  93.018         	|  22.136         	|
run_validate_evaluation            	|  12.359         	|1              	|  12.359         	|  2.9412         	|
evaluation_step_and_end            	|  0.24717        	|48             	|  11.864         	|  2.8234         	|
validation_step                    	|  0.24712        	|48             	|  11.862         	|  2.8227         	|
get_train_batch                    	|  0.0081357      	|1063           	|  8.6482         	|  2.0581         	|
fetch_next_train_batch             	|  0.0081173      	|1063           	|  8.6287         	|  2.0534         	|
zero_grad                          	|  0.0013138      	|1000           	|  1.3138         	|  0.31265        	|
on_train_batch_end                 	|  0.0012179      	|1000           	|  1.2179         	|  0.28982        	|
training_batch_to_device           	|  0.00022879     	|1000           	|  0.22879        	|  0.054446       	|
get_validate_batch                 	|  0.0043991      	|49             	|  0.21556        	|  0.051297       	|
fetch_next_validate_batch          	|  0.0043836      	|49             	|  0.2148         	|  0.051116       	|
on_train_batch_start               	|  9.1517e-05     	|1000           	|  0.091517       	|  0.021779       	|
on_train_end                       	|  0.025543       	|1              	|  0.025543       	|  0.0060786      	|
on_pretrain_routine_start          	|  0.02275        	|1              	|  0.02275        	|  0.005414       	|
on_train_epoch_start               	|  0.00029401     	|63             	|  0.018523       	|  0.004408       	|
on_train_epoch_end                 	|  0.00028395     	|63             	|  0.017889       	|  0.004257       	|
on_batch_start                     	|  1.6587e-05     	|1000           	|  0.016587       	|  0.0039472      	|
on_before_zero_grad                	|  1.3618e-05     	|1000           	|  0.013618       	|  0.0032408      	|
on_batch_end                       	|  1.3103e-05     	|1000           	|  0.013103       	|  0.0031181      	|
on_after_backward                  	|  1.1357e-05     	|1000           	|  0.011357       	|  0.0027026      	|
on_before_backward                 	|  1.0579e-05     	|1000           	|  0.010579       	|  0.0025175      	|
on_before_optimizer_step           	|  1.0499e-05     	|1000           	|  0.010499       	|  0.0024984      	|
evaluation_batch_to_device         	|  0.00021052     	|48             	|  0.010105       	|  0.0024047      	|
configure_optimizers               	|  0.0087758      	|1              	|  0.0087758      	|  0.0020884      	|
training_step_end                  	|  5.7342e-06     	|1000           	|  0.0057342      	|  0.0013646      	|
on_validation_model_eval           	|  0.0039082      	|1              	|  0.0039082      	|  0.00093004     	|
on_validation_batch_end            	|  2.0441e-05     	|48             	|  0.00098118     	|  0.0002335      	|
on_train_start                     	|  0.00078289     	|1              	|  0.00078289     	|  0.00018631     	|
on_epoch_start                     	|  1.1015e-05     	|64             	|  0.00070498     	|  0.00016777     	|
on_epoch_end                       	|  1.0144e-05     	|64             	|  0.00064924     	|  0.0001545      	|
on_validation_batch_start          	|  1.032e-05      	|48             	|  0.00049537     	|  0.00011789     	|
on_validation_start                	|  0.00031513     	|1              	|  0.00031513     	|  7.4993e-05     	|
validation_step_end                	|  5.2736e-06     	|48             	|  0.00025313     	|  6.0239e-05     	|
on_validation_end                  	|  0.0001131      	|1              	|  0.0001131      	|  2.6915e-05     	|
on_before_accelerator_backend_setup	|  1.9156e-05     	|2              	|  3.8312e-05     	|  9.1172e-06     	|
on_configure_sharded_model         	|  1.7412e-05     	|2              	|  3.4825e-05     	|  8.2874e-06     	|
teardown                           	|  1.4903e-05     	|2              	|  2.9806e-05     	|  7.093e-06      	|
setup                              	|  1.2864e-05     	|2              	|  2.5728e-05     	|  6.1226e-06     	|
on_fit_end                         	|  2.0599e-05     	|1              	|  2.0599e-05     	|  4.902e-06      	|
on_validation_epoch_end            	|  1.7483e-05     	|1              	|  1.7483e-05     	|  4.1605e-06     	|
on_fit_start                       	|  1.6691e-05     	|1              	|  1.6691e-05     	|  3.972e-06      	|
prepare_data                       	|  6.427e-06      	|2              	|  1.2854e-05     	|  3.0589e-06     	|
on_val_dataloader                  	|  6.0966e-06     	|2              	|  1.2193e-05     	|  2.9016e-06     	|
on_pretrain_routine_end            	|  1.0299e-05     	|1              	|  1.0299e-05     	|  2.4509e-06     	|
on_validation_epoch_start          	|  1.023e-05      	|1              	|  1.023e-05      	|  2.4345e-06     	|
configure_sharded_model            	|  4.7041e-06     	|2              	|  9.4081e-06     	|  2.2389e-06     	|
configure_callbacks                	|  4.679e-06      	|2              	|  9.3579e-06     	|  2.2269e-06     	|
on_train_dataloader                	|  5.43e-06       	|1              	|  5.43e-06       	|  1.2922e-06     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1619.7         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.428         	|63             	|  1539.0         	|  95.017         	|
run_training_batch                 	|  0.18799        	|8000           	|  1503.9         	|  92.851         	|
training_step_and_backward         	|  0.17759        	|8000           	|  1420.7         	|  87.717         	|
model_forward                      	|  0.092423       	|8000           	|  739.38         	|  45.649         	|
training_step                      	|  0.092289       	|8000           	|  738.31         	|  45.583         	|
backward                           	|  0.084986       	|8000           	|  679.89         	|  41.976         	|
optimizer_step_with_closure_0      	|  0.23803        	|1000           	|  238.03         	|  14.696         	|
run_validate_evaluation            	|  37.258         	|1              	|  37.258         	|  2.3003         	|
evaluation_step_and_end            	|  0.096572       	|377            	|  36.408         	|  2.2478         	|
validation_step                    	|  0.096523       	|377            	|  36.389         	|  2.2467         	|
get_train_batch                    	|  0.002273       	|8063           	|  18.327         	|  1.1315         	|
fetch_next_train_batch             	|  0.0022565      	|8063           	|  18.194         	|  1.1233         	|
on_train_batch_end                 	|  0.001072       	|8000           	|  8.576          	|  0.52948        	|
training_batch_to_device           	|  0.00021461     	|8000           	|  1.7169         	|  0.106          	|
zero_grad                          	|  0.0013352      	|1000           	|  1.3352         	|  0.082435       	|
on_train_batch_start               	|  8.9603e-05     	|8000           	|  0.71683        	|  0.044257       	|
get_validate_batch                 	|  0.001608       	|378            	|  0.60784        	|  0.037528       	|
fetch_next_validate_batch          	|  0.0015947      	|378            	|  0.60278        	|  0.037216       	|
on_batch_start                     	|  1.4395e-05     	|8000           	|  0.11516        	|  0.0071102      	|
on_before_backward                 	|  1.3032e-05     	|8000           	|  0.10426        	|  0.0064368      	|
on_batch_end                       	|  1.2687e-05     	|8000           	|  0.1015         	|  0.0062666      	|
on_after_backward                  	|  1.1345e-05     	|8000           	|  0.090757       	|  0.0056034      	|
evaluation_batch_to_device         	|  0.00020612     	|377            	|  0.077709       	|  0.0047977      	|
training_step_end                  	|  5.6314e-06     	|8000           	|  0.045052       	|  0.0027815      	|
on_train_end                       	|  0.025634       	|1              	|  0.025634       	|  0.0015826      	|
on_pretrain_routine_start          	|  0.022993       	|1              	|  0.022993       	|  0.0014196      	|
on_train_epoch_start               	|  0.00029185     	|63             	|  0.018387       	|  0.0011352      	|
on_train_epoch_end                 	|  0.00028305     	|63             	|  0.017832       	|  0.0011009      	|
on_before_zero_grad                	|  1.4211e-05     	|1000           	|  0.014211       	|  0.00087741     	|
on_before_optimizer_step           	|  1.1439e-05     	|1000           	|  0.011439       	|  0.00070622     	|
configure_optimizers               	|  0.0088648      	|1              	|  0.0088648      	|  0.00054731     	|
on_validation_batch_end            	|  1.7617e-05     	|377            	|  0.0066415      	|  0.00041005     	|
on_validation_model_eval           	|  0.0052287      	|1              	|  0.0052287      	|  0.00032282     	|
on_validation_batch_start          	|  9.435e-06      	|377            	|  0.003557       	|  0.00021961     	|
validation_step_end                	|  4.8094e-06     	|377            	|  0.0018131      	|  0.00011194     	|
on_train_start                     	|  0.00073376     	|1              	|  0.00073376     	|  4.5302e-05     	|
on_epoch_start                     	|  1.0823e-05     	|64             	|  0.00069267     	|  4.2766e-05     	|
on_epoch_end                       	|  9.6453e-06     	|64             	|  0.0006173      	|  3.8112e-05     	|
on_validation_start                	|  0.00035396     	|1              	|  0.00035396     	|  2.1853e-05     	|
on_validation_end                  	|  0.00010724     	|1              	|  0.00010724     	|  6.6208e-06     	|
on_before_accelerator_backend_setup	|  1.9622e-05     	|2              	|  3.9243e-05     	|  2.4229e-06     	|
teardown                           	|  1.6866e-05     	|2              	|  3.3732e-05     	|  2.0826e-06     	|
setup                              	|  1.3751e-05     	|2              	|  2.7501e-05     	|  1.6979e-06     	|
on_configure_sharded_model         	|  1.3561e-05     	|2              	|  2.7121e-05     	|  1.6745e-06     	|
on_fit_end                         	|  2.2882e-05     	|1              	|  2.2882e-05     	|  1.4127e-06     	|
on_validation_epoch_end            	|  1.7121e-05     	|1              	|  1.7121e-05     	|  1.0571e-06     	|
on_fit_start                       	|  1.6191e-05     	|1              	|  1.6191e-05     	|  9.9964e-07     	|
on_val_dataloader                  	|  6.3219e-06     	|2              	|  1.2644e-05     	|  7.8063e-07     	|
configure_callbacks                	|  6.2219e-06     	|2              	|  1.2444e-05     	|  7.6829e-07     	|
on_pretrain_routine_end            	|  1.2013e-05     	|1              	|  1.2013e-05     	|  7.4168e-07     	|
prepare_data                       	|  5.886e-06      	|2              	|  1.1772e-05     	|  7.268e-07      	|
on_validation_epoch_start          	|  9.628e-06      	|1              	|  9.628e-06      	|  5.9443e-07     	|
configure_sharded_model            	|  4.7639e-06     	|2              	|  9.5279e-06     	|  5.8825e-07     	|
on_train_dataloader                	|  4.3779e-06     	|1              	|  4.3779e-06     	|  2.7029e-07     	|
