Skip to content

# Struct ExtraInfo

- Defined in File trtorch.h

## Nested Relationships

### Nested Types

- Class ExtraInfo::DataType

- Class ExtraInfo::DeviceType

- Struct ExtraInfo::InputRange

## Struct Documentation

struct `trtorch `::`` `ExtraInfo`

  Settings data structure for TRTorch compilation

  Public Types

  enum `EngineCapability`

    Emum for selecting engine capability

    Values:

    enumerator `kDEFAULT`
    enumerator `kSAFE_GPU`
    enumerator `kSAFE_DLA`

  Public Functions

  `ExtraInfo` ( std::vector< InputRange > input_ranges )

    Construct a new Extra Info object from input ranges. Each entry in the vector represents a input and should be provided in call order.

    Use this constructor if you want to use dynamic shape

    Parameters

      - `input_ranges` :

  `ExtraInfo` ( std::vector<std::vector<int64_t>> fixed_sizes )

    Construct a new Extra Info object Convienence constructor to set fixed input size from vectors describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.

    Parameters

      - `fixed_sizes` :

  `ExtraInfo` ( std::vector<c10::ArrayRef<int64_t>> fixed_sizes )

    Construct a new Extra Info object Convienence constructor to set fixed input size from c10::ArrayRef’s (the output of tensor.sizes()) describing size of input tensors. Each entry in the vector represents a input and should be provided in call order.

    Parameters

      - `fixed_sizes` :

  Public Members

  std::vector< InputRange > `input_ranges`

    Sizes for inputs to engine, can either be a single size or a range defined by Min, Optimal, Max sizes

    Order is should match call order

  DataType `op_precision` = DataType ::kFloat

    Default operating precision for the engine

  bool `refit` = false

    Build a refitable engine

  bool `debug` = false

    Build a debugable engine

  bool `strict_types` = false

    Restrict operating type to only set default operation precision (op_precision)

  bool `allow_gpu_fallback` = true

    (Only used when targeting DLA (device)) Lets engine run layers on GPU if they are not supported on DLA

  DeviceType `device` = DeviceType ::kGPU

    Target device type

  EngineCapability `capability` = EngineCapability :: kDEFAULT

    Sets the restrictions for the engine (CUDA Safety)

  uint64_t `num_min_timing_iters` = 2

    Number of minimization timing iterations used to select kernels

  uint64_t `num_avg_timing_iters` = 1

    Number of averaging timing iterations used to select kernels

  uint64_t `workspace_size` = 0

    Maximum size of workspace given to TensorRT

  uint64_t `max_batch_size` = 0

    Maximum batch size (must be >= 1 to be set, 0 means not set)

  nvinfer1::IInt8Calibrator * `ptq_calibrator` = nullptr

    Calibration dataloaders for each input for post training quantizatiom

  class `DataType`

    Supported Data Types that can be used with TensorRT engines

    This class is compatable with c10::DataTypes (but will check for TRT support) so there should not be a reason that you need to use this type explictly.

    Public Types

    enum `Value`

      Underlying enum class to support the DataType Class

      In the case that you need to use the DataType class itself, interface using this enum vs. normal instatination

      ex. trtorch::DataType type = DataType::kFloat ;

      Values:

      enumerator `kFloat`

        FP32.

      enumerator `kHalf`

        FP16.

      enumerator `kChar`

        INT8.

    Public Functions

    `DataType` ( ) = default

      Construct a new Data Type object.

    constexpr `DataType` ( Value t )

      DataType constructor from enum.

    `DataType` ( c10::ScalarType t )

      Construct a new Data Type object from torch type enums.

      Parameters

        - `t` :

    `operator Value` ( ) const

      Get the enum value of the DataType object.

      Return

        Value

    `operator bool` ( ) = delete
    constexpr bool `operator==` ( DataType other ) const

      Comparision operator for DataType .

      Return

        true

      Return

        false

      Parameters

        - `other` :

    constexpr bool `operator!=` ( DataType other ) const

      Comparision operator for DataType .

      Return

        true

      Return

        false

      Parameters

        - `other` :

  class `DeviceType`

    Supported Device Types that can be used with TensorRT engines

    This class is compatable with c10::DeviceTypes (but will check for TRT support) but the only applicable value is at::kCUDA, which maps to DeviceType::kGPU

    To use the DataType class itself, interface using the enum vs. normal instatination

    ex. trtorch::DeviceType type = DeviceType::kGPU ;

    Public Types

    enum `Value`

      Underlying enum class to support the DeviceType Class

      In the case that you need to use the DeviceType class itself, interface using this enum vs. normal instatination

      ex. trtorch::DeviceType type = DeviceType::kGPU ;

      Values:

      enumerator `kGPU`

        Target GPU to run engine.

      enumerator `kDLA`

        Target DLA to run engine.

    Public Functions

    `DeviceType` ( ) = default

      Construct a new Device Type object.

    constexpr `DeviceType` ( Value t )

      Construct a new Device Type object from internal enum.

    `DeviceType` ( c10::DeviceType t )

      Construct a new Device Type object from torch device enums Note: The only valid value is torch::kCUDA (torch::kCPU is not supported)

      Parameters

        - `t` :

    `operator Value` ( ) const

      Get the internal value from the Device object.

      Return

        Value

    `operator bool` ( ) = delete
    constexpr bool `operator==` ( DeviceType other ) const

      Comparison operator for DeviceType .

      Return

        true

      Return

        false

      Parameters

        - `other` :

    constexpr bool `operator!=` ( DeviceType other ) const

      Comparison operator for DeviceType .

      Return

        true

      Return

        false

      Parameters

        - `other` :

  struct `InputRange`

    A struct to hold an input range (used by TensorRT Optimization profile)

    This struct can either hold a single vector representing an input shape, signifying a static input shape or a set of three input shapes representing the min, optiminal and max input shapes allowed for the engine.

    Public Functions

    `InputRange` ( std::vector<int64_t> opt )

      Construct a new Input Range object for static input size from vector.

      Parameters

        - `opt` :

    `InputRange` ( c10::ArrayRef<int64_t> opt )

      Construct a new Input Range object static input size from c10::ArrayRef (the type produced by tensor.sizes())

      Parameters

        - `opt` :

    `InputRange` ( std::vector<int64_t> min , std::vector<int64_t> opt , std::vector<int64_t> max )

      Construct a new Input Range object dynamic input size from vectors for min, opt, and max supported sizes.

      Parameters

        - `min` :

        - `opt` :

        - `max` :

    `InputRange` ( c10::ArrayRef<int64_t> min , c10::ArrayRef<int64_t> opt , c10::ArrayRef<int64_t> max )

      Construct a new Input Range object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes.

      Parameters

        - `min` :

        - `opt` :

        - `max` :

    Public Members

    std::vector<int64_t> `min`

      Minimum acceptable input size into the engine.

    std::vector<int64_t> `opt`

      Optimal input size into the engine (gets best performace)

    std::vector<int64_t> `max`

      Maximum acceptable input size into the engine.