VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  419.24         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.7477         	|63             	|  362.11         	|  86.371         	|
run_training_batch                 	|  0.33539        	|1000           	|  335.39         	|  79.999         	|
optimizer_step_with_closure_0      	|  0.33283        	|1000           	|  332.83         	|  79.389         	|
training_step_and_backward         	|  0.24858        	|1000           	|  248.58         	|  59.292         	|
model_forward                      	|  0.15458        	|1000           	|  154.58         	|  36.87          	|
training_step                      	|  0.15444        	|1000           	|  154.44         	|  36.837         	|
backward                           	|  0.092668       	|1000           	|  92.668         	|  22.103         	|
run_validate_evaluation            	|  12.373         	|1              	|  12.373         	|  2.9512         	|
evaluation_step_and_end            	|  0.2472         	|48             	|  11.866         	|  2.8302         	|
validation_step                    	|  0.24715        	|48             	|  11.863         	|  2.8296         	|
get_train_batch                    	|  0.0081321      	|1063           	|  8.6445         	|  2.0619         	|
fetch_next_train_batch             	|  0.0081135      	|1063           	|  8.6247         	|  2.0572         	|
zero_grad                          	|  0.0013101      	|1000           	|  1.3101         	|  0.31248        	|
on_train_batch_end                 	|  0.0011792      	|1000           	|  1.1792         	|  0.28126        	|
get_validate_batch                 	|  0.0046389      	|49             	|  0.22731        	|  0.054218       	|
fetch_next_validate_batch          	|  0.0046231      	|49             	|  0.22653        	|  0.054034       	|
training_batch_to_device           	|  0.00022517     	|1000           	|  0.22517        	|  0.05371        	|
on_train_batch_start               	|  9.4063e-05     	|1000           	|  0.094063       	|  0.022436       	|
on_train_end                       	|  0.025655       	|1              	|  0.025655       	|  0.0061192      	|
on_pretrain_routine_start          	|  0.022736       	|1              	|  0.022736       	|  0.0054231      	|
on_train_epoch_start               	|  0.00030499     	|63             	|  0.019214       	|  0.004583       	|
on_train_epoch_end                 	|  0.00028987     	|63             	|  0.018262       	|  0.0043559      	|
on_batch_start                     	|  1.6342e-05     	|1000           	|  0.016342       	|  0.0038981      	|
on_batch_end                       	|  1.3704e-05     	|1000           	|  0.013704       	|  0.0032688      	|
on_before_zero_grad                	|  1.3512e-05     	|1000           	|  0.013512       	|  0.0032229      	|
on_after_backward                  	|  1.1416e-05     	|1000           	|  0.011416       	|  0.0027229      	|
on_before_backward                 	|  1.1121e-05     	|1000           	|  0.011121       	|  0.0026525      	|
evaluation_batch_to_device         	|  0.00022567     	|48             	|  0.010832       	|  0.0025837      	|
on_before_optimizer_step           	|  1.0522e-05     	|1000           	|  0.010522       	|  0.0025098      	|
configure_optimizers               	|  0.0082292      	|1              	|  0.0082292      	|  0.0019629      	|
training_step_end                  	|  5.9225e-06     	|1000           	|  0.0059225      	|  0.0014126      	|
on_validation_model_eval           	|  0.0042546      	|1              	|  0.0042546      	|  0.0010148      	|
on_validation_batch_end            	|  2.1708e-05     	|48             	|  0.001042       	|  0.00024853     	|
on_train_start                     	|  0.00073773     	|1              	|  0.00073773     	|  0.00017597     	|
on_epoch_start                     	|  1.0676e-05     	|64             	|  0.00068328     	|  0.00016298     	|
on_epoch_end                       	|  1.0353e-05     	|64             	|  0.00066258     	|  0.00015804     	|
on_validation_batch_start          	|  1.0848e-05     	|48             	|  0.0005207      	|  0.0001242      	|
on_validation_start                	|  0.00032497     	|1              	|  0.00032497     	|  7.7513e-05     	|
validation_step_end                	|  5.2894e-06     	|48             	|  0.00025389     	|  6.0559e-05     	|
on_validation_end                  	|  0.00011131     	|1              	|  0.00011131     	|  2.655e-05      	|
on_before_accelerator_backend_setup	|  1.8736e-05     	|2              	|  3.7471e-05     	|  8.9378e-06     	|
teardown                           	|  1.5689e-05     	|2              	|  3.1379e-05     	|  7.4846e-06     	|
on_configure_sharded_model         	|  1.3696e-05     	|2              	|  2.7391e-05     	|  6.5334e-06     	|
setup                              	|  1.345e-05      	|2              	|  2.69e-05       	|  6.4163e-06     	|
on_fit_end                         	|  2.139e-05      	|1              	|  2.139e-05      	|  5.102e-06      	|
on_validation_epoch_end            	|  1.9586e-05     	|1              	|  1.9586e-05     	|  4.6717e-06     	|
on_fit_start                       	|  1.6311e-05     	|1              	|  1.6311e-05     	|  3.8906e-06     	|
on_pretrain_routine_end            	|  1.4157e-05     	|1              	|  1.4157e-05     	|  3.3768e-06     	|
prepare_data                       	|  6.006e-06      	|2              	|  1.2012e-05     	|  2.8651e-06     	|
on_val_dataloader                  	|  5.856e-06      	|2              	|  1.1712e-05     	|  2.7936e-06     	|
on_validation_epoch_start          	|  1.1051e-05     	|1              	|  1.1051e-05     	|  2.6359e-06     	|
configure_sharded_model            	|  5.2e-06        	|2              	|  1.04e-05       	|  2.4806e-06     	|
configure_callbacks                	|  4.7896e-06     	|2              	|  9.5791e-06     	|  2.2849e-06     	|
on_train_dataloader                	|  4.3879e-06     	|1              	|  4.3879e-06     	|  1.0466e-06     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1604.9         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.197         	|63             	|  1524.4         	|  94.988         	|
run_training_batch                 	|  0.18619        	|8000           	|  1489.5         	|  92.813         	|
training_step_and_backward         	|  0.17602        	|8000           	|  1408.2         	|  87.744         	|
model_forward                      	|  0.092195       	|8000           	|  737.56         	|  45.958         	|
training_step                      	|  0.092064       	|8000           	|  736.51         	|  45.892         	|
backward                           	|  0.083643       	|8000           	|  669.15         	|  41.695         	|
optimizer_step_with_closure_0      	|  0.2339         	|1000           	|  233.9          	|  14.574         	|
run_validate_evaluation            	|  37.246         	|1              	|  37.246         	|  2.3208         	|
evaluation_step_and_end            	|  0.096548       	|377            	|  36.399         	|  2.268          	|
validation_step                    	|  0.096499       	|377            	|  36.38          	|  2.2669         	|
get_train_batch                    	|  0.0022549      	|8063           	|  18.181         	|  1.1329         	|
fetch_next_train_batch             	|  0.0022385      	|8063           	|  18.049         	|  1.1246         	|
on_train_batch_end                 	|  0.0010788      	|8000           	|  8.6301         	|  0.53775        	|
training_batch_to_device           	|  0.00021122     	|8000           	|  1.6898         	|  0.10529        	|
zero_grad                          	|  0.0013202      	|1000           	|  1.3202         	|  0.08226        	|
on_train_batch_start               	|  8.9638e-05     	|8000           	|  0.7171         	|  0.044683       	|
get_validate_batch                 	|  0.0016033      	|378            	|  0.60604        	|  0.037763       	|
fetch_next_validate_batch          	|  0.00159        	|378            	|  0.60103        	|  0.03745        	|
on_batch_start                     	|  1.4344e-05     	|8000           	|  0.11475        	|  0.0071501      	|
on_batch_end                       	|  1.3165e-05     	|8000           	|  0.10532        	|  0.0065625      	|
on_before_backward                 	|  1.2588e-05     	|8000           	|  0.1007         	|  0.0062749      	|
on_after_backward                  	|  1.1586e-05     	|8000           	|  0.092688       	|  0.0057755      	|
evaluation_batch_to_device         	|  0.00021135     	|377            	|  0.079678       	|  0.0049648      	|
training_step_end                  	|  5.9807e-06     	|8000           	|  0.047846       	|  0.0029813      	|
on_train_end                       	|  0.025971       	|1              	|  0.025971       	|  0.0016183      	|
on_pretrain_routine_start          	|  0.022093       	|1              	|  0.022093       	|  0.0013766      	|
on_train_epoch_start               	|  0.00030965     	|63             	|  0.019508       	|  0.0012156      	|
on_train_epoch_end                 	|  0.00027804     	|63             	|  0.017517       	|  0.0010915      	|
on_before_zero_grad                	|  1.4834e-05     	|1000           	|  0.014834       	|  0.00092429     	|
on_before_optimizer_step           	|  1.2039e-05     	|1000           	|  0.012039       	|  0.00075017     	|
configure_optimizers               	|  0.0078151      	|1              	|  0.0078151      	|  0.00048696     	|
on_validation_batch_end            	|  1.8805e-05     	|377            	|  0.0070893      	|  0.00044174     	|
on_validation_model_eval           	|  0.0039751      	|1              	|  0.0039751      	|  0.00024769     	|
on_validation_batch_start          	|  9.7779e-06     	|377            	|  0.0036863      	|  0.00022969     	|
validation_step_end                	|  5.1094e-06     	|377            	|  0.0019263      	|  0.00012003     	|
on_train_start                     	|  0.00070094     	|1              	|  0.00070094     	|  4.3676e-05     	|
on_epoch_start                     	|  1.0499e-05     	|64             	|  0.00067191     	|  4.1867e-05     	|
on_epoch_end                       	|  1.0315e-05     	|64             	|  0.00066018     	|  4.1136e-05     	|
on_validation_start                	|  0.00034888     	|1              	|  0.00034888     	|  2.1739e-05     	|
on_validation_end                  	|  0.00011212     	|1              	|  0.00011212     	|  6.9862e-06     	|
on_before_accelerator_backend_setup	|  1.8394e-05     	|2              	|  3.6788e-05     	|  2.2923e-06     	|
teardown                           	|  1.6601e-05     	|2              	|  3.3202e-05     	|  2.0688e-06     	|
setup                              	|  1.3485e-05     	|2              	|  2.697e-05      	|  1.6805e-06     	|
on_configure_sharded_model         	|  1.3044e-05     	|2              	|  2.6088e-05     	|  1.6256e-06     	|
on_fit_end                         	|  2.2943e-05     	|1              	|  2.2943e-05     	|  1.4296e-06     	|
on_validation_epoch_end            	|  1.8324e-05     	|1              	|  1.8324e-05     	|  1.1418e-06     	|
on_fit_start                       	|  1.657e-05      	|1              	|  1.657e-05      	|  1.0325e-06     	|
on_val_dataloader                  	|  6.7626e-06     	|2              	|  1.3525e-05     	|  8.4276e-07     	|
prepare_data                       	|  6.5221e-06     	|2              	|  1.3044e-05     	|  8.1278e-07     	|
on_pretrain_routine_end            	|  1.2733e-05     	|1              	|  1.2733e-05     	|  7.934e-07      	|
configure_callbacks                	|  5.5999e-06     	|2              	|  1.12e-05       	|  6.9787e-07     	|
configure_sharded_model            	|  4.7035e-06     	|2              	|  9.4071e-06     	|  5.8616e-07     	|
on_validation_epoch_start          	|  9.337e-06      	|1              	|  9.337e-06      	|  5.8179e-07     	|
on_train_dataloader                	|  4.3879e-06     	|1              	|  4.3879e-06     	|  2.7341e-07     	|
