Table of Contents
Shortcuts

# Torch-TensorRT

## Ahead-of-time compilation of TorchScript / PyTorch JIT for NVIDIA GPUs

Torch-TensorRT is a compiler for PyTorch/TorchScript, targeting NVIDIA GPUs via NVIDIA’s TensorRT Deep Learning Optimizer and Runtime. Unlike PyTorch’s Just-In-Time (JIT) compiler, Torch-TensorRT is an Ahead-of-Time (AOT) compiler, meaning that before you deploy your TorchScript code, you go through an explicit compile step to convert a standard TorchScript program into an module targeting a TensorRT engine. Torch-TensorRT operates as a PyTorch extention and compiles modules that integrate into the JIT runtime seamlessly. After compilation using the optimized graph should feel no different than running a TorchScript module. You also have access to TensorRT’s suite of configurations at compile time, so you are able to specify operating precision (FP32/FP16/INT8) and other settings for your module.

More Information / System Architecture:

- GTC 2020 Talk

## Getting Started

- Installation

- Using Torch-TensorRT in Python

- Using Torch-TensorRT in C++

## Tutorials

- Creating a TorchScript Module

- Torch-TensorRT (FX Frontend) User Guide

- Post Training Quantization (PTQ)

- Deploying Torch-TensorRT Programs

- Serving a Torch-TensorRT model with Triton

- Using Torch-TensorRT Directly From PyTorch

- DLA

- Example notebooks

## Python API Documenation

- torch_tensorrt

- torch_tensorrt.logging

- torch_tensorrt.ptq

- torch_tensorrt.ts

- torch_tensorrt.fx

## C++ API Documenation

- Namespace torch_tensorrt

- Namespace torch_tensorrt::logging

- Namespace torch_tensorrt::ptq

- Namespace torch_tensorrt::torchscript

## CLI Documentation

- torchtrtc

## Contributor Documentation

- System Overview

- Writing Converters

- Useful Links for Torch-TensorRT Development

## Indices

- Operators Supported

- Index

- Search Page

## Docs

Access comprehensive developer documentation for PyTorch

View Docs

## Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

## Resources

Find development resources and get your questions answered

View Resources

To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. As the current maintainers of this site, Facebook’s Cookies Policy applies. Learn more, including about available controls: Cookies Policy.