VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  508.9          	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  2.5799         	|167            	|  430.85         	|  84.662         	|
run_training_batch                 	|  0.37717        	|1000           	|  377.17         	|  74.113         	|
optimizer_step_with_closure_0      	|  0.37447        	|1000           	|  374.47         	|  73.583         	|
training_step_and_backward         	|  0.29443        	|1000           	|  294.43         	|  57.855         	|
model_forward                      	|  0.17101        	|1000           	|  171.01         	|  33.604         	|
training_step                      	|  0.17087        	|1000           	|  170.87         	|  33.576         	|
backward                           	|  0.12206        	|1000           	|  122.06         	|  23.985         	|
run_validate_evaluation            	|  34.831         	|1              	|  34.831         	|  6.8444         	|
evaluation_step_and_end            	|  0.27374        	|125            	|  34.218         	|  6.7238         	|
validation_step                    	|  0.27369        	|125            	|  34.211         	|  6.7225         	|
get_train_batch                    	|  0.014118       	|1167           	|  16.476         	|  3.2375         	|
fetch_next_train_batch             	|  0.014095       	|1167           	|  16.449         	|  3.2322         	|
zero_grad                          	|  0.0013243      	|1000           	|  1.3243         	|  0.26022        	|
on_train_batch_end                 	|  0.0012329      	|1000           	|  1.2329         	|  0.24227        	|
get_validate_batch                 	|  0.0026264      	|126            	|  0.33092        	|  0.065027       	|
fetch_next_validate_batch          	|  0.002612       	|126            	|  0.32911        	|  0.06467        	|
training_batch_to_device           	|  0.00031746     	|1000           	|  0.31746        	|  0.062381       	|
on_train_batch_start               	|  9.5269e-05     	|1000           	|  0.095269       	|  0.01872        	|
on_train_epoch_start               	|  0.00027663     	|167            	|  0.046198       	|  0.0090779      	|
on_train_epoch_end                 	|  0.00026923     	|167            	|  0.044962       	|  0.008835       	|
evaluation_batch_to_device         	|  0.00022791     	|125            	|  0.028488       	|  0.005598       	|
on_train_end                       	|  0.024873       	|1              	|  0.024873       	|  0.0048877      	|
on_pretrain_routine_start          	|  0.022858       	|1              	|  0.022858       	|  0.0044917      	|
on_batch_start                     	|  2.02e-05       	|1000           	|  0.0202         	|  0.0039693      	|
on_before_zero_grad                	|  1.576e-05      	|1000           	|  0.01576        	|  0.0030969      	|
on_batch_end                       	|  1.4023e-05     	|1000           	|  0.014023       	|  0.0027555      	|
on_after_backward                  	|  1.1376e-05     	|1000           	|  0.011376       	|  0.0022353      	|
on_before_optimizer_step           	|  1.1222e-05     	|1000           	|  0.011222       	|  0.0022051      	|
on_before_backward                 	|  1.0773e-05     	|1000           	|  0.010773       	|  0.002117       	|
configure_optimizers               	|  0.0083372      	|1              	|  0.0083372      	|  0.0016383      	|
training_step_end                  	|  6.6043e-06     	|1000           	|  0.0066043      	|  0.0012977      	|
on_validation_model_eval           	|  0.0040608      	|1              	|  0.0040608      	|  0.00079794     	|
on_validation_batch_end            	|  2.1446e-05     	|125            	|  0.0026808      	|  0.00052678     	|
on_epoch_start                     	|  1.0886e-05     	|168            	|  0.0018288      	|  0.00035936     	|
on_epoch_end                       	|  1.0691e-05     	|168            	|  0.0017961      	|  0.00035294     	|
on_validation_batch_start          	|  1.041e-05      	|125            	|  0.0013012      	|  0.00025569     	|
on_train_start                     	|  0.00071732     	|1              	|  0.00071732     	|  0.00014095     	|
validation_step_end                	|  5.6158e-06     	|125            	|  0.00070198     	|  0.00013794     	|
on_validation_start                	|  0.0003115      	|1              	|  0.0003115      	|  6.1211e-05     	|
on_validation_end                  	|  0.00011173     	|1              	|  0.00011173     	|  2.1955e-05     	|
on_before_accelerator_backend_setup	|  1.8094e-05     	|2              	|  3.6188e-05     	|  7.111e-06      	|
teardown                           	|  1.5404e-05     	|2              	|  3.0808e-05     	|  6.0538e-06     	|
on_configure_sharded_model         	|  1.4292e-05     	|2              	|  2.8584e-05     	|  5.6168e-06     	|
setup                              	|  1.3931e-05     	|2              	|  2.7862e-05     	|  5.4749e-06     	|
on_fit_end                         	|  2.1841e-05     	|1              	|  2.1841e-05     	|  4.2918e-06     	|
on_validation_epoch_end            	|  1.7644e-05     	|1              	|  1.7644e-05     	|  3.4671e-06     	|
on_fit_start                       	|  1.5589e-05     	|1              	|  1.5589e-05     	|  3.0632e-06     	|
on_val_dataloader                  	|  6.1464e-06     	|2              	|  1.2293e-05     	|  2.4156e-06     	|
prepare_data                       	|  5.6455e-06     	|2              	|  1.1291e-05     	|  2.2187e-06     	|
configure_callbacks                	|  5.2895e-06     	|2              	|  1.0579e-05     	|  2.0788e-06     	|
configure_sharded_model            	|  5.145e-06      	|2              	|  1.029e-05      	|  2.022e-06      	|
on_pretrain_routine_end            	|  1.0029e-05     	|1              	|  1.0029e-05     	|  1.9707e-06     	|
on_validation_epoch_start          	|  9.197e-06      	|1              	|  9.197e-06      	|  1.8072e-06     	|
on_train_dataloader                	|  3.848e-06      	|1              	|  3.848e-06      	|  7.5613e-07     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1680.9         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  9.2014         	|167            	|  1536.6         	|  91.418         	|
run_training_batch                 	|  0.18619        	|8000           	|  1489.5         	|  88.615         	|
training_step_and_backward         	|  0.17601        	|8000           	|  1408.0         	|  83.768         	|
model_forward                      	|  0.093492       	|8000           	|  747.93         	|  44.496         	|
training_step                      	|  0.093357       	|8000           	|  746.85         	|  44.432         	|
backward                           	|  0.082326       	|8000           	|  658.61         	|  39.182         	|
optimizer_step_with_closure_0      	|  0.23433        	|1000           	|  234.33         	|  13.941         	|
run_validate_evaluation            	|  101.47         	|1              	|  101.47         	|  6.0367         	|
evaluation_step_and_end            	|  0.099489       	|1000           	|  99.489         	|  5.9188         	|
validation_step                    	|  0.099439       	|1000           	|  99.439         	|  5.9159         	|
get_train_batch                    	|  0.003356       	|8167           	|  27.408         	|  1.6306         	|
fetch_next_train_batch             	|  0.0033386      	|8167           	|  27.267         	|  1.6222         	|
on_train_batch_end                 	|  0.0010811      	|8000           	|  8.6487         	|  0.51453        	|
training_batch_to_device           	|  0.00021736     	|8000           	|  1.7389         	|  0.10345        	|
get_validate_batch                 	|  0.0014543      	|1001           	|  1.4558         	|  0.086606       	|
fetch_next_validate_batch          	|  0.0014407      	|1001           	|  1.4421         	|  0.085796       	|
zero_grad                          	|  0.001354       	|1000           	|  1.354          	|  0.080554       	|
on_train_batch_start               	|  9.2301e-05     	|8000           	|  0.73841        	|  0.04393        	|
evaluation_batch_to_device         	|  0.00020345     	|1000           	|  0.20345        	|  0.012104       	|
on_batch_start                     	|  1.5415e-05     	|8000           	|  0.12332        	|  0.0073366      	|
on_batch_end                       	|  1.3437e-05     	|8000           	|  0.10749        	|  0.006395       	|
on_before_backward                 	|  1.3211e-05     	|8000           	|  0.10569        	|  0.0062875      	|
on_after_backward                  	|  1.2434e-05     	|8000           	|  0.09947        	|  0.0059177      	|
training_step_end                  	|  6.1804e-06     	|8000           	|  0.049444       	|  0.0029415      	|
on_train_epoch_start               	|  0.0002788      	|167            	|  0.046559       	|  0.0027699      	|
on_train_epoch_end                 	|  0.00025743     	|167            	|  0.042991       	|  0.0025577      	|
on_train_end                       	|  0.025706       	|1              	|  0.025706       	|  0.0015293      	|
on_pretrain_routine_start          	|  0.022781       	|1              	|  0.022781       	|  0.0013553      	|
on_validation_batch_end            	|  1.7751e-05     	|1000           	|  0.017751       	|  0.0010561      	|
on_before_zero_grad                	|  1.6791e-05     	|1000           	|  0.016791       	|  0.00099894     	|
on_before_optimizer_step           	|  1.3802e-05     	|1000           	|  0.013802       	|  0.00082114     	|
on_validation_batch_start          	|  1.042e-05      	|1000           	|  0.01042        	|  0.0006199      	|
configure_optimizers               	|  0.0086641      	|1              	|  0.0086641      	|  0.00051545     	|
validation_step_end                	|  4.5764e-06     	|1000           	|  0.0045764      	|  0.00027226     	|
on_validation_model_eval           	|  0.0045032      	|1              	|  0.0045032      	|  0.00026791     	|
on_epoch_start                     	|  1.0975e-05     	|168            	|  0.0018437      	|  0.00010969     	|
on_epoch_end                       	|  1.0149e-05     	|168            	|  0.001705       	|  0.00010143     	|
on_train_start                     	|  0.00075062     	|1              	|  0.00075062     	|  4.4656e-05     	|
on_validation_start                	|  0.00035027     	|1              	|  0.00035027     	|  2.0839e-05     	|
on_validation_end                  	|  0.00010862     	|1              	|  0.00010862     	|  6.4622e-06     	|
on_before_accelerator_backend_setup	|  1.9236e-05     	|2              	|  3.8472e-05     	|  2.2888e-06     	|
teardown                           	|  1.4958e-05     	|2              	|  2.9916e-05     	|  1.7798e-06     	|
setup                              	|  1.4777e-05     	|2              	|  2.9555e-05     	|  1.7583e-06     	|
on_configure_sharded_model         	|  1.3365e-05     	|2              	|  2.673e-05      	|  1.5902e-06     	|
on_fit_end                         	|  2.1901e-05     	|1              	|  2.1901e-05     	|  1.3029e-06     	|
on_fit_start                       	|  1.7543e-05     	|1              	|  1.7543e-05     	|  1.0437e-06     	|
on_validation_epoch_end            	|  1.7202e-05     	|1              	|  1.7202e-05     	|  1.0234e-06     	|
prepare_data                       	|  6.241e-06      	|2              	|  1.2482e-05     	|  7.4259e-07     	|
on_val_dataloader                  	|  5.9961e-06     	|2              	|  1.1992e-05     	|  7.1344e-07     	|
configure_callbacks                	|  5.816e-06      	|2              	|  1.1632e-05     	|  6.9202e-07     	|
on_validation_epoch_start          	|  1.05e-05       	|1              	|  1.05e-05       	|  6.2467e-07     	|
configure_sharded_model            	|  5.2101e-06     	|2              	|  1.042e-05      	|  6.1992e-07     	|
on_pretrain_routine_end            	|  1.0079e-05     	|1              	|  1.0079e-05     	|  5.9962e-07     	|
on_train_dataloader                	|  4.3381e-06     	|1              	|  4.3381e-06     	|  2.5808e-07     	|
