FIT Profiler Report

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  1683.1         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  26.35          	|63             	|  1660.0         	|  98.629         	|
run_training_batch                 	|  0.20312        	|8000           	|  1625.0         	|  96.547         	|
training_step_and_backward         	|  0.19185        	|8000           	|  1534.8         	|  91.187         	|
model_forward                      	|  0.10823        	|8000           	|  865.82         	|  51.442         	|
training_step                      	|  0.10809        	|8000           	|  864.76         	|  51.379         	|
backward                           	|  0.083439       	|8000           	|  667.52         	|  39.66          	|
optimizer_step_with_closure_0      	|  0.25001        	|1000           	|  250.01         	|  14.854         	|
get_train_batch                    	|  0.0022664      	|8063           	|  18.274         	|  1.0857         	|
fetch_next_train_batch             	|  0.0022498      	|8063           	|  18.14          	|  1.0778         	|
on_train_batch_end                 	|  0.0010881      	|8000           	|  8.705          	|  0.5172         	|
training_batch_to_device           	|  0.00021488     	|8000           	|  1.719          	|  0.10213        	|
zero_grad                          	|  0.0012985      	|1000           	|  1.2985         	|  0.077152       	|
on_train_batch_start               	|  9.0542e-05     	|8000           	|  0.72433        	|  0.043036       	|
on_batch_start                     	|  1.5051e-05     	|8000           	|  0.12041        	|  0.0071541      	|
on_batch_end                       	|  1.3352e-05     	|8000           	|  0.10681        	|  0.0063463      	|
on_before_backward                 	|  1.2452e-05     	|8000           	|  0.099616       	|  0.0059186      	|
on_after_backward                  	|  1.1651e-05     	|8000           	|  0.093206       	|  0.0055378      	|
training_step_end                  	|  5.6629e-06     	|8000           	|  0.045303       	|  0.0026916      	|
on_train_end                       	|  0.025918       	|1              	|  0.025918       	|  0.0015399      	|
on_pretrain_routine_start          	|  0.02297        	|1              	|  0.02297        	|  0.0013648      	|
on_train_epoch_start               	|  0.00030862     	|63             	|  0.019443       	|  0.0011552      	|
on_train_epoch_end                 	|  0.00029676     	|63             	|  0.018696       	|  0.0011108      	|
on_before_zero_grad                	|  1.6288e-05     	|1000           	|  0.016288       	|  0.00096771     	|
on_before_optimizer_step           	|  1.2102e-05     	|1000           	|  0.012102       	|  0.00071902     	|
configure_optimizers               	|  0.0086555      	|1              	|  0.0086555      	|  0.00051426     	|
on_train_start                     	|  0.0007867      	|1              	|  0.0007867      	|  4.6741e-05     	|
on_epoch_start                     	|  1.0566e-05     	|63             	|  0.00066565     	|  3.9549e-05     	|
on_epoch_end                       	|  1.0557e-05     	|63             	|  0.00066511     	|  3.9517e-05     	|
setup                              	|  2.2131e-05     	|1              	|  2.2131e-05     	|  1.3149e-06     	|
on_fit_end                         	|  2.0909e-05     	|1              	|  2.0909e-05     	|  1.2423e-06     	|
on_before_accelerator_backend_setup	|  2.0709e-05     	|1              	|  2.0709e-05     	|  1.2304e-06     	|
on_fit_start                       	|  1.7733e-05     	|1              	|  1.7733e-05     	|  1.0536e-06     	|
on_configure_sharded_model         	|  1.6311e-05     	|1              	|  1.6311e-05     	|  9.6912e-07     	|
teardown                           	|  1.03e-05       	|1              	|  1.03e-05       	|  6.1196e-07     	|
on_pretrain_routine_end            	|  1.0089e-05     	|1              	|  1.0089e-05     	|  5.9943e-07     	|
prepare_data                       	|  8.3358e-06     	|1              	|  8.3358e-06     	|  4.9526e-07     	|
configure_callbacks                	|  5.74e-06       	|1              	|  5.74e-06       	|  3.4104e-07     	|
configure_sharded_model            	|  5.6711e-06     	|1              	|  5.6711e-06     	|  3.3694e-07     	|
on_val_dataloader                  	|  4.5781e-06     	|1              	|  4.5781e-06     	|  2.7201e-07     	|
on_train_dataloader                	|  4.448e-06      	|1              	|  4.448e-06      	|  2.6427e-07     	|
