Table of Contents
Shortcuts

Note

Go to the end to download the full example code

cross runtime compilation limitations: The cross compile and saved model can only be loaded in Windows, it can no longer be loaded in Linux The cross compile and saved model can only be loaded in the same Compute Capability as the Linux which it was cross compiled (for example, if the model was cross compiled in Linux with GeForceRTX 4080 which has Compute Capability of 8.9, It cannot be loaded in Windows with GeForceRTX 3080 which has Compute Capability of 8.6)

# Cross runtime compilation for windows example

Compile and save the Resnet Model using Torch-TensorRT in Linux:

python examples/dynamo/cross_runtime_compilation_for_windows.py –path trt_resnet.ep

Load the Resnet Model saved in Windows:

python examples/dynamo/cross_runtime_compilation_for_windows.py –path trt_resnet.ep –load True

## Imports and Model Definition

    import argparse
    import platform

    import torch
    import torch_tensorrt as torchtrt
    import torchvision.models as models

    PARSER = argparse.ArgumentParser(
        description="Cross runtime comilation for windows example: Resnet Model"
    )
    PARSER.add_argument(
        "--load", default=False, type=bool, required=False, help="Load the model in Windows"
    )
    PARSER.add_argument(
        "--path",
        type=str,
        required=True,
        help="Path to the saved model file",
    )

    args = PARSER.parse_args()
    torch.manual_seed(0)
    model = models.resnet18().eval().cuda()
    input = torch.rand((1, 3, 224, 224)).to("cuda")
    inputs = [input]

According to the argument, it is either cross compile and save resnet model for windows in Linux or load the saved resnet model in Windows ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    if args.load:
        # load the saved model in Windows
        if platform.system() != "Windows" or platform.machine() != "AMD64":
            raise ValueError(
                "cross runtime compiled model for windows can only be loaded in Windows system"
            )
        loaded_model = torchtrt.load_cross_compiled_exported_program(args.path).module()
        print(f"model has been successfully loaded from ${args.path}")
        # inference
        trt_output = loaded_model(input)
        print(f"inference result: {trt_output}")
    else:
        if platform.system() != "Linux" or platform.architecture()[0] != "64bit":
            raise ValueError(
                "cross runtime compiled model for windows can only be compiled in Linux system"
            )
        compile_spec = {
            "debug": True,
            "min_block_size": 1,
        }
        torchtrt.cross_compile_for_windows(
            model, file_path=args.path, inputs=inputs, **compile_spec
        )
        print(
            f"model has been successfully cross compiled and saved in Linux to {args.path}"
        )

Total running time of the script: ( 0 minutes 0.000 seconds)

Gallery generated by Sphinx-Gallery

## Docs

Access comprehensive developer documentation for PyTorch

View Docs

## Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

## Resources

Find development resources and get your questions answered

View Resources

To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. As the current maintainers of this site, Facebook’s Cookies Policy applies. Learn more, including about available controls: Cookies Policy.