VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  421.1          	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  5.7744         	|63             	|  363.79         	|  86.391         	|
run_training_batch                 	|  0.33724        	|1000           	|  337.24         	|  80.085         	|
optimizer_step_with_closure_0      	|  0.33462        	|1000           	|  334.62         	|  79.465         	|
training_step_and_backward         	|  0.25054        	|1000           	|  250.54         	|  59.496         	|
model_forward                      	|  0.15547        	|1000           	|  155.47         	|  36.921         	|
training_step                      	|  0.15533        	|1000           	|  155.33         	|  36.888         	|
backward                           	|  0.093743       	|1000           	|  93.743         	|  22.262         	|
run_validate_evaluation            	|  12.37          	|1              	|  12.37          	|  2.9375         	|
evaluation_step_and_end            	|  0.24721        	|48             	|  11.866         	|  2.8179         	|
validation_step                    	|  0.24716        	|48             	|  11.863         	|  2.8173         	|
get_train_batch                    	|  0.0080987      	|1063           	|  8.6089         	|  2.0444         	|
fetch_next_train_batch             	|  0.008079       	|1063           	|  8.588          	|  2.0394         	|
zero_grad                          	|  0.001297       	|1000           	|  1.297          	|  0.308          	|
on_train_batch_end                 	|  0.0011894      	|1000           	|  1.1894         	|  0.28246        	|
training_batch_to_device           	|  0.0002287      	|1000           	|  0.2287         	|  0.05431        	|
get_validate_batch                 	|  0.0044408      	|49             	|  0.2176         	|  0.051675       	|
fetch_next_validate_batch          	|  0.004426       	|49             	|  0.21688        	|  0.051503       	|
on_train_batch_start               	|  9.0328e-05     	|1000           	|  0.090328       	|  0.021451       	|
on_train_end                       	|  0.025295       	|1              	|  0.025295       	|  0.0060069      	|
on_pretrain_routine_start          	|  0.022709       	|1              	|  0.022709       	|  0.0053928      	|
on_train_epoch_end                 	|  0.0002935      	|63             	|  0.01849        	|  0.004391       	|
on_train_epoch_start               	|  0.00028114     	|63             	|  0.017712       	|  0.0042061      	|
on_batch_start                     	|  1.6163e-05     	|1000           	|  0.016163       	|  0.0038384      	|
on_batch_end                       	|  1.3882e-05     	|1000           	|  0.013882       	|  0.0032967      	|
on_before_zero_grad                	|  1.3251e-05     	|1000           	|  0.013251       	|  0.0031467      	|
on_after_backward                  	|  1.2576e-05     	|1000           	|  0.012576       	|  0.0029866      	|
on_before_optimizer_step           	|  1.0999e-05     	|1000           	|  0.010999       	|  0.0026121      	|
evaluation_batch_to_device         	|  0.00022304     	|48             	|  0.010706       	|  0.0025423      	|
on_before_backward                 	|  1.0427e-05     	|1000           	|  0.010427       	|  0.0024762      	|
configure_optimizers               	|  0.0085046      	|1              	|  0.0085046      	|  0.0020196      	|
training_step_end                  	|  5.7113e-06     	|1000           	|  0.0057113      	|  0.0013563      	|
on_validation_model_eval           	|  0.0046642      	|1              	|  0.0046642      	|  0.0011076      	|
on_validation_batch_end            	|  2.0625e-05     	|48             	|  0.00099002     	|  0.00023511     	|
on_train_start                     	|  0.00074127     	|1              	|  0.00074127     	|  0.00017603     	|
on_epoch_start                     	|  1.044e-05      	|64             	|  0.00066816     	|  0.00015867     	|
on_epoch_end                       	|  1.032e-05      	|64             	|  0.00066049     	|  0.00015685     	|
on_validation_batch_start          	|  1.0989e-05     	|48             	|  0.00052747     	|  0.00012526     	|
on_validation_start                	|  0.00033068     	|1              	|  0.00033068     	|  7.8528e-05     	|
validation_step_end                	|  4.8285e-06     	|48             	|  0.00023177     	|  5.5039e-05     	|
on_validation_end                  	|  0.00011075     	|1              	|  0.00011075     	|  2.63e-05       	|
on_before_accelerator_backend_setup	|  1.8921e-05     	|2              	|  3.7841e-05     	|  8.9863e-06     	|
teardown                           	|  1.584e-05      	|2              	|  3.168e-05      	|  7.5232e-06     	|
on_configure_sharded_model         	|  1.3886e-05     	|2              	|  2.7772e-05     	|  6.5952e-06     	|
setup                              	|  1.2959e-05     	|2              	|  2.5918e-05     	|  6.1549e-06     	|
on_fit_end                         	|  2.093e-05      	|1              	|  2.093e-05      	|  4.9703e-06     	|
on_fit_start                       	|  1.8104e-05     	|1              	|  1.8104e-05     	|  4.2992e-06     	|
on_validation_epoch_end            	|  1.7763e-05     	|1              	|  1.7763e-05     	|  4.2183e-06     	|
on_val_dataloader                  	|  5.8311e-06     	|2              	|  1.1662e-05     	|  2.7695e-06     	|
prepare_data                       	|  5.3955e-06     	|2              	|  1.0791e-05     	|  2.5626e-06     	|
configure_sharded_model            	|  5.2695e-06     	|2              	|  1.0539e-05     	|  2.5028e-06     	|
on_pretrain_routine_end            	|  9.838e-06      	|1              	|  9.838e-06      	|  2.3363e-06     	|
on_validation_epoch_start          	|  9.738e-06      	|1              	|  9.738e-06      	|  2.3125e-06     	|
configure_callbacks                	|  4.4785e-06     	|2              	|  8.957e-06      	|  2.1271e-06     	|
on_train_dataloader                	|  4.439e-06      	|1              	|  4.439e-06      	|  1.0542e-06     	|
VALIDATE Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
-------------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1612.5         	|  100 %          	|
-------------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  24.315         	|63             	|  1531.8         	|  94.995         	|
run_training_batch                 	|  0.18712        	|8000           	|  1497.0         	|  92.833         	|
training_step_and_backward         	|  0.17699        	|8000           	|  1415.9         	|  87.808         	|
model_forward                      	|  0.092263       	|8000           	|  738.1          	|  45.772         	|
training_step                      	|  0.09213        	|8000           	|  737.04         	|  45.707         	|
backward                           	|  0.084545       	|8000           	|  676.36         	|  41.944         	|
optimizer_step_with_closure_0      	|  0.23547        	|1000           	|  235.47         	|  14.602         	|
run_validate_evaluation            	|  37.149         	|1              	|  37.149         	|  2.3038         	|
evaluation_step_and_end            	|  0.096315       	|377            	|  36.311         	|  2.2518         	|
validation_step                    	|  0.096264       	|377            	|  36.292         	|  2.2506         	|
get_train_batch                    	|  0.0022547      	|8063           	|  18.18          	|  1.1274         	|
fetch_next_train_batch             	|  0.0022381      	|8063           	|  18.046         	|  1.1191         	|
on_train_batch_end                 	|  0.0010714      	|8000           	|  8.5709         	|  0.53151        	|
training_batch_to_device           	|  0.00021226     	|8000           	|  1.6981         	|  0.1053         	|
zero_grad                          	|  0.0013381      	|1000           	|  1.3381         	|  0.082982       	|
on_train_batch_start               	|  9.0133e-05     	|8000           	|  0.72107        	|  0.044716       	|
get_validate_batch                 	|  0.0015817      	|378            	|  0.59787        	|  0.037076       	|
fetch_next_validate_batch          	|  0.0015678      	|378            	|  0.59261        	|  0.03675        	|
on_batch_start                     	|  1.4281e-05     	|8000           	|  0.11425        	|  0.0070851      	|
on_batch_end                       	|  1.2861e-05     	|8000           	|  0.10289        	|  0.0063804      	|
on_before_backward                 	|  1.2553e-05     	|8000           	|  0.10043        	|  0.0062278      	|
on_after_backward                  	|  1.1249e-05     	|8000           	|  0.089995       	|  0.0055809      	|
evaluation_batch_to_device         	|  0.00020347     	|377            	|  0.07671        	|  0.0047571      	|
training_step_end                  	|  5.4052e-06     	|8000           	|  0.043242       	|  0.0026816      	|
on_train_end                       	|  0.02553        	|1              	|  0.02553        	|  0.0015832      	|
on_pretrain_routine_start          	|  0.023166       	|1              	|  0.023166       	|  0.0014366      	|
on_train_epoch_start               	|  0.00029749     	|63             	|  0.018742       	|  0.0011622      	|
on_train_epoch_end                 	|  0.00026828     	|63             	|  0.016902       	|  0.0010481      	|
on_before_zero_grad                	|  1.4791e-05     	|1000           	|  0.014791       	|  0.00091722     	|
on_before_optimizer_step           	|  1.2213e-05     	|1000           	|  0.012213       	|  0.00075735     	|
configure_optimizers               	|  0.0077136      	|1              	|  0.0077136      	|  0.00047835     	|
on_validation_batch_end            	|  1.8179e-05     	|377            	|  0.0068535      	|  0.00042501     	|
on_validation_model_eval           	|  0.0040682      	|1              	|  0.0040682      	|  0.00025228     	|
on_validation_batch_start          	|  9.9239e-06     	|377            	|  0.0037413      	|  0.00023201     	|
validation_step_end                	|  4.6646e-06     	|377            	|  0.0017585      	|  0.00010905     	|
on_epoch_start                     	|  1.0975e-05     	|64             	|  0.00070237     	|  4.3557e-05     	|
on_train_start                     	|  0.00069778     	|1              	|  0.00069778     	|  4.3272e-05     	|
on_epoch_end                       	|  9.9008e-06     	|64             	|  0.00063365     	|  3.9295e-05     	|
on_validation_start                	|  0.00031745     	|1              	|  0.00031745     	|  1.9686e-05     	|
on_validation_end                  	|  0.00011416     	|1              	|  0.00011416     	|  7.0797e-06     	|
on_before_accelerator_backend_setup	|  1.8685e-05     	|2              	|  3.737e-05      	|  2.3175e-06     	|
teardown                           	|  1.6721e-05     	|2              	|  3.3443e-05     	|  2.0739e-06     	|
on_configure_sharded_model         	|  1.4011e-05     	|2              	|  2.8022e-05     	|  1.7378e-06     	|
setup                              	|  1.343e-05      	|2              	|  2.686e-05      	|  1.6657e-06     	|
on_fit_end                         	|  2.0238e-05     	|1              	|  2.0238e-05     	|  1.255e-06      	|
on_validation_epoch_end            	|  1.7793e-05     	|1              	|  1.7793e-05     	|  1.1034e-06     	|
on_fit_start                       	|  1.595e-05      	|1              	|  1.595e-05      	|  9.8911e-07     	|
on_val_dataloader                  	|  6.2165e-06     	|2              	|  1.2433e-05     	|  7.7101e-07     	|
prepare_data                       	|  5.9304e-06     	|2              	|  1.1861e-05     	|  7.3554e-07     	|
on_pretrain_routine_end            	|  1.1822e-05     	|1              	|  1.1822e-05     	|  7.3313e-07     	|
configure_callbacks                	|  5.5599e-06     	|2              	|  1.112e-05      	|  6.8958e-07     	|
configure_sharded_model            	|  5.2151e-06     	|2              	|  1.043e-05      	|  6.4681e-07     	|
on_validation_epoch_start          	|  8.8061e-06     	|1              	|  8.8061e-06     	|  5.461e-07      	|
on_train_dataloader                	|  4.3879e-06     	|1              	|  4.3879e-06     	|  2.7211e-07     	|
