VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  420.43         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.7643         	|63             	|  363.15         	|  86.375         	|
run_training_batch                 	|  0.33645        	|1000           	|  336.45         	|  80.024         	|
optimizer_step_with_closure_0      	|  0.3337         	|1000           	|  333.7          	|  79.37          	|
training_step_and_backward         	|  0.2502         	|1000           	|  250.2          	|  59.51          	|
model_forward                      	|  0.15521        	|1000           	|  155.21         	|  36.916         	|
training_step                      	|  0.15507        	|1000           	|  155.07         	|  36.883         	|
backward                           	|  0.093715       	|1000           	|  93.715         	|  22.29          	|
run_validate_evaluation            	|  12.451         	|1              	|  12.451         	|  2.9615         	|
evaluation_step_and_end            	|  0.24944        	|48             	|  11.973         	|  2.8478         	|
validation_step                    	|  0.24938        	|48             	|  11.97          	|  2.8472         	|
get_train_batch                    	|  0.0081096      	|1063           	|  8.6205         	|  2.0504         	|
fetch_next_train_batch             	|  0.0080906      	|1063           	|  8.6003         	|  2.0456         	|
zero_grad                          	|  0.0012541      	|1000           	|  1.2541         	|  0.2983         	|
on_train_batch_end                 	|  0.0012065      	|1000           	|  1.2065         	|  0.28696        	|
training_batch_to_device           	|  0.00022977     	|1000           	|  0.22977        	|  0.054652       	|
get_validate_batch                 	|  0.0040369      	|49             	|  0.19781        	|  0.047048       	|
fetch_next_validate_batch          	|  0.004022       	|49             	|  0.19708        	|  0.046875       	|
on_train_batch_start               	|  9.0475e-05     	|1000           	|  0.090475       	|  0.02152        	|
on_train_end                       	|  0.025775       	|1              	|  0.025775       	|  0.0061306      	|
on_pretrain_routine_start          	|  0.022163       	|1              	|  0.022163       	|  0.0052715      	|
on_train_epoch_end                 	|  0.00028691     	|63             	|  0.018075       	|  0.0042992      	|
on_train_epoch_start               	|  0.00027999     	|63             	|  0.017639       	|  0.0041956      	|
on_batch_start                     	|  1.5485e-05     	|1000           	|  0.015485       	|  0.0036831      	|
on_before_zero_grad                	|  1.3695e-05     	|1000           	|  0.013695       	|  0.0032575      	|
on_batch_end                       	|  1.2813e-05     	|1000           	|  0.012813       	|  0.0030476      	|
on_after_backward                  	|  1.1921e-05     	|1000           	|  0.011921       	|  0.0028354      	|
evaluation_batch_to_device         	|  0.0002205      	|48             	|  0.010584       	|  0.0025174      	|
on_before_optimizer_step           	|  1.0199e-05     	|1000           	|  0.010199       	|  0.0024259      	|
on_before_backward                 	|  1.015e-05      	|1000           	|  0.01015        	|  0.0024141      	|
configure_optimizers               	|  0.0083304      	|1              	|  0.0083304      	|  0.0019814      	|
training_step_end                  	|  5.7417e-06     	|1000           	|  0.0057417      	|  0.0013657      	|
on_validation_model_eval           	|  0.0045104      	|1              	|  0.0045104      	|  0.0010728      	|
on_validation_batch_end            	|  2.1187e-05     	|48             	|  0.001017       	|  0.00024189     	|
on_train_start                     	|  0.0007662      	|1              	|  0.0007662      	|  0.00018224     	|
on_epoch_end                       	|  1.1148e-05     	|64             	|  0.0007135      	|  0.00016971     	|
on_epoch_start                     	|  1.0288e-05     	|64             	|  0.00065844     	|  0.00015661     	|
on_validation_batch_start          	|  1.0694e-05     	|48             	|  0.0005133      	|  0.00012209     	|
on_validation_start                	|  0.00032957     	|1              	|  0.00032957     	|  7.8388e-05     	|
validation_step_end                	|  5.5061e-06     	|48             	|  0.00026429     	|  6.2862e-05     	|
on_validation_end                  	|  0.00011233     	|1              	|  0.00011233     	|  2.6719e-05     	|
on_before_accelerator_backend_setup	|  1.8935e-05     	|2              	|  3.7871e-05     	|  9.0076e-06     	|
teardown                           	|  1.5249e-05     	|2              	|  3.0498e-05     	|  7.254e-06      	|
on_configure_sharded_model         	|  1.3861e-05     	|2              	|  2.7722e-05     	|  6.5937e-06     	|
setup                              	|  1.2884e-05     	|2              	|  2.5768e-05     	|  6.129e-06      	|
on_fit_end                         	|  2.2062e-05     	|1              	|  2.2062e-05     	|  5.2475e-06     	|
on_validation_epoch_end            	|  1.9897e-05     	|1              	|  1.9897e-05     	|  4.7325e-06     	|
on_fit_start                       	|  1.6882e-05     	|1              	|  1.6882e-05     	|  4.0154e-06     	|
on_val_dataloader                  	|  6.2615e-06     	|2              	|  1.2523e-05     	|  2.9786e-06     	|
prepare_data                       	|  6.031e-06      	|2              	|  1.2062e-05     	|  2.869e-06      	|
on_pretrain_routine_end            	|  1.0429e-05     	|1              	|  1.0429e-05     	|  2.4805e-06     	|
configure_sharded_model            	|  5.0295e-06     	|2              	|  1.0059e-05     	|  2.3925e-06     	|
on_validation_epoch_start          	|  9.258e-06      	|1              	|  9.258e-06      	|  2.202e-06      	|
configure_callbacks                	|  4.6139e-06     	|2              	|  9.2279e-06     	|  2.1949e-06     	|
on_train_dataloader                	|  4.338e-06      	|1              	|  4.338e-06      	|  1.0318e-06     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1616.3         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.378         	|63             	|  1535.8         	|  95.02          	|
run_training_batch                 	|  0.1876         	|8000           	|  1500.8         	|  92.857         	|
training_step_and_backward         	|  0.17748        	|8000           	|  1419.8         	|  87.846         	|
model_forward                      	|  0.092543       	|8000           	|  740.34         	|  45.805         	|
training_step                      	|  0.092411       	|8000           	|  739.29         	|  45.74          	|
backward                           	|  0.084757       	|8000           	|  678.05         	|  41.952         	|
optimizer_step_with_closure_0      	|  0.23584        	|1000           	|  235.84         	|  14.591         	|
run_validate_evaluation            	|  37.359         	|1              	|  37.359         	|  2.3114         	|
evaluation_step_and_end            	|  0.096855       	|377            	|  36.514         	|  2.2592         	|
validation_step                    	|  0.096805       	|377            	|  36.495         	|  2.258          	|
get_train_batch                    	|  0.0022642      	|8063           	|  18.256         	|  1.1295         	|
fetch_next_train_batch             	|  0.0022473      	|8063           	|  18.12          	|  1.1211         	|
on_train_batch_end                 	|  0.0010762      	|8000           	|  8.6098         	|  0.5327         	|
training_batch_to_device           	|  0.00021053     	|8000           	|  1.6842         	|  0.1042         	|
zero_grad                          	|  0.0012951      	|1000           	|  1.2951         	|  0.080127       	|
on_train_batch_start               	|  8.8908e-05     	|8000           	|  0.71127        	|  0.044007       	|
get_validate_batch                 	|  0.0015897      	|378            	|  0.60089        	|  0.037177       	|
fetch_next_validate_batch          	|  0.0015756      	|378            	|  0.59559        	|  0.03685        	|
on_batch_start                     	|  1.4603e-05     	|8000           	|  0.11682        	|  0.0072278      	|
on_before_backward                 	|  1.3056e-05     	|8000           	|  0.10445        	|  0.0064621      	|
on_batch_end                       	|  1.2706e-05     	|8000           	|  0.10165        	|  0.0062891      	|
on_after_backward                  	|  1.1915e-05     	|8000           	|  0.095318       	|  0.0058974      	|
evaluation_batch_to_device         	|  0.00020677     	|377            	|  0.077954       	|  0.0048231      	|
training_step_end                  	|  5.8759e-06     	|8000           	|  0.047007       	|  0.0029084      	|
on_train_end                       	|  0.025634       	|1              	|  0.025634       	|  0.001586       	|
on_pretrain_routine_start          	|  0.022209       	|1              	|  0.022209       	|  0.0013741      	|
on_train_epoch_start               	|  0.00029243     	|63             	|  0.018423       	|  0.0011399      	|
on_train_epoch_end                 	|  0.00028278     	|63             	|  0.017815       	|  0.0011023      	|
on_before_zero_grad                	|  1.4931e-05     	|1000           	|  0.014931       	|  0.00092377     	|
on_before_optimizer_step           	|  1.1494e-05     	|1000           	|  0.011494       	|  0.00071112     	|
configure_optimizers               	|  0.0088595      	|1              	|  0.0088595      	|  0.00054815     	|
on_validation_batch_end            	|  1.89e-05       	|377            	|  0.0071251      	|  0.00044084     	|
on_validation_model_eval           	|  0.0040449      	|1              	|  0.0040449      	|  0.00025026     	|
on_validation_batch_start          	|  9.9828e-06     	|377            	|  0.0037635      	|  0.00023285     	|
validation_step_end                	|  4.9603e-06     	|377            	|  0.00187        	|  0.0001157      	|
on_train_start                     	|  0.00073988     	|1              	|  0.00073988     	|  4.5777e-05     	|
on_epoch_start                     	|  1.0796e-05     	|64             	|  0.00069093     	|  4.2748e-05     	|
on_epoch_end                       	|  1.0261e-05     	|64             	|  0.00065668     	|  4.0629e-05     	|
on_validation_start                	|  0.00032779     	|1              	|  0.00032779     	|  2.0281e-05     	|
on_validation_end                  	|  0.00011503     	|1              	|  0.00011503     	|  7.1167e-06     	|
on_before_accelerator_backend_setup	|  1.9627e-05     	|2              	|  3.9253e-05     	|  2.4286e-06     	|
teardown                           	|  1.4958e-05     	|2              	|  2.9917e-05     	|  1.851e-06      	|
on_configure_sharded_model         	|  1.3846e-05     	|2              	|  2.7692e-05     	|  1.7133e-06     	|
setup                              	|  1.3445e-05     	|2              	|  2.689e-05      	|  1.6637e-06     	|
on_fit_end                         	|  2.2542e-05     	|1              	|  2.2542e-05     	|  1.3947e-06     	|
on_fit_start                       	|  1.8164e-05     	|1              	|  1.8164e-05     	|  1.1238e-06     	|
on_validation_epoch_end            	|  1.7132e-05     	|1              	|  1.7132e-05     	|  1.06e-06       	|
on_pretrain_routine_end            	|  1.2914e-05     	|1              	|  1.2914e-05     	|  7.9899e-07     	|
prepare_data                       	|  6.4163e-06     	|2              	|  1.2833e-05     	|  7.9397e-07     	|
on_val_dataloader                  	|  6.0964e-06     	|2              	|  1.2193e-05     	|  7.5438e-07     	|
configure_callbacks                	|  5.4005e-06     	|2              	|  1.0801e-05     	|  6.6827e-07     	|
configure_sharded_model            	|  4.895e-06      	|2              	|  9.7901e-06     	|  6.0572e-07     	|
on_validation_epoch_start          	|  9.5181e-06     	|1              	|  9.5181e-06     	|  5.8889e-07     	|
on_train_dataloader                	|  4.3379e-06     	|1              	|  4.3379e-06     	|  2.6839e-07     	|
