Skip to content

# trtorchc

`trtorchc` is a CLI application for using the TRTorch compiler. It serves as an easy way to compile a TorchScript Module with TRTorch from the command-line to quickly check support or as part of a deployment pipeline. All basic features of the compiler are supported including post training quantization (though you must already have a calibration cache file to use the PTQ feature). The compiler can output two formats, either a TorchScript program with the TensorRT engine embedded or the TensorRT engine itself as a PLAN file.

All that is required to run the program after compilation is for C++ linking against `libtrtorch.so` or in Python importing the trtorch package. All other aspects of using compiled modules are identical to standard TorchScript. Load with `torch.jit.load()` and run like you would run any other module.

    trtorchc [input_file_path] [output_file_path]
        [input_shapes...] {OPTIONS}

        TRTorch is a compiler for TorchScript, it will compile and optimize
        TorchScript programs to run on NVIDIA GPUs using TensorRT

    OPTIONS:

        -h, --help                        Display this help menu
        Verbiosity of the compiler
            -v, --verbose                     Dumps debugging information about the
                                            compilation process onto the console
            -w, --warnings                    Disables warnings generated during
                                            compilation onto the console (warnings
                                            are on by default)
            --info                            Dumps info messages generated during
                                            compilation onto the console
        --build-debuggable-engine         Creates a debuggable engine
        --use-strict-types                Restrict operating type to only use set
                                            default operation precision
                                            (op_precision)
        --allow-gpu-fallback              (Only used when targeting DLA
                                            (device-type)) Lets engine run layers on
                                            GPU if they are not supported on DLA
        -p[precision],
        --default-op-precision=[precision]
                                            Default operating precision for the
                                            engine (Int8 requires a
                                            calibration-cache argument) [ float |
                                            float32 | f32 | half | float16 | f16 |
                                            int8 | i8 ] (default: float)
        -d[type], --device-type=[type]    The type of device the engine should be
                                            built for [ gpu | dla ] (default: gpu)
        --engine-capability=[capability]  The type of device the engine should be
                                            built for [ default | safe_gpu |
                                            safe_dla ]
        --calibration-cache-file=[file_path]
                                            Path to calibration cache file to use
                                            for post training quantization
        --num-min-timing-iter=[num_iters] Number of minimization timing iterations
                                            used to select kernels
        --num-avg-timing-iters=[num_iters]
                                            Number of averaging timing iterations
                                            used to select kernels
        --workspace-size=[workspace_size] Maximum size of workspace given to
                                            TensorRT
        --max-batch-size=[max_batch_size] Maximum batch size (must be >= 1 to be
                                            set, 0 means not set)
        -t[threshold],
        --threshold=[threshold]           Maximum acceptable numerical deviation
                                            from standard torchscript output
                                            (default 2e-5)
        --save-engine                     Instead of compiling a full a
                                            TorchScript program, save the created
                                            engine to the path specified as the
                                            output path
        input_file_path                   Path to input TorchScript file
        output_file_path                  Path for compiled TorchScript (or
                                            TensorRT engine) file
        input_shapes...                   Sizes for inputs to engine, can either
                                            be a single size or a range defined by
                                            Min, Optimal, Max sizes, e.g.
                                            "(N,..,C,H,W)"
                                            "[(MIN_N,..,MIN_C,MIN_H,MIN_W);(OPT_N,..,OPT_C,OPT_H,OPT_W);(MAX_N,..,MAX_C,MAX_H,MAX_W)]"
        "--" can be used to terminate flag options and force all following
        arguments to be treated as positional options

e.g.

    trtorchc tests/modules/ssd_traced.jit.pt ssd_trt.ts "[(1,3,300,300); (1,3,512,512); (1, 3, 1024, 1024)]" -p f16