🐛 [Bug] RuntimeError: upper bound and larger bound inconsistent with step sign

[hturki]

Bug Description

The code below results in the following error:

RuntimeError: upper bound and larger bound inconsistent with step sign

Note that if one doesn't freeze the script module before trying to compile, a different error occurs:

RuntimeError: Method 'forward' is not defined.

To Reproduce

Steps to reproduce the behavior:

Try running the following code:

import torch
import torch_tensorrt
import torch.nn as nn


class TestEmbedding(nn.Module):
    def __init__(self, N_freqs):
        super(TestEmbedding, self).__init__()
        self.freq_bands = 2 ** torch.linspace(0, N_freqs - 1, N_freqs)

    def forward2(self, x):
        out = [x]
        for freq in self.freq_bands:
            out += [torch.sin(freq * x), torch.cos(freq * x)]

        return torch.cat(out, -1)


class TestNeRF(nn.Module):
    def __init__(self, D=8, W=256, N_freq_xyz=10, skips=[4], rgb_dim=3):
        super(TestNeRF, self).__init__()
        self.D = D
        self.W = W

        self.embedding_xyz = TestEmbedding(N_freq_xyz)
        in_channels_xyz = 3 + 3 * N_freq_xyz * 2

        self.skips = skips

        xyz_encodings = []

        # xyz encoding layers
        for i in range(D):
            if i == 0:
                layer = nn.Linear(in_channels_xyz, W)
            elif i in skips:
                layer = nn.Linear(W + in_channels_xyz, W)
            else:
                layer = nn.Linear(W, W)
            layer = nn.Sequential(layer, nn.ReLU(True))
            xyz_encodings.append(layer)

        self.xyz_encodings = nn.ModuleList(xyz_encodings)

        # output layers
        self.sigma = nn.Linear(W, 1)
        self.rgb = nn.Linear(W, rgb_dim)

    def forward(self, x):
        input_xyz = self.embedding_xyz.forward2(x[:, :3])
        xyz_ = input_xyz
        for i, xyz_encoding in enumerate(self.xyz_encodings):
            if i in self.skips:
                xyz_ = torch.cat([input_xyz, xyz_], -1)
            xyz_ = xyz_encoding(xyz_)

        sigma = self.sigma(xyz_)

        rgb = self.rgb(xyz_)
        return torch.cat([rgb, sigma], -1)


sub_model = torch.jit.trace(TestNeRF(), torch.ones(1, 3))
print('before compile', sub_model(torch.ones(1, 3)))

trt_ts_module = torch_tensorrt.compile(sub_model,
                                       inputs=[torch_tensorrt.Input(
                                           min_shape=[1, 3, 224, 224],
                                           opt_shape=[1, 3, 512, 512],
                                           max_shape=[1, 3, 1024, 1024],
                                           dtype=torch.float)
                                       ],
                                       enabled_precisions={torch.float}
                                       )

print('after compile', trt_ts_module(torch.ones(1, 3)))

Expected behavior

Code compiles

Environment

Build information about the TRTorch compiler can be found by turning on debug messages

• TRTorch Version (e.g. 0.2.0): 1.0.0
• PyTorch Version (e.g. 1.0): 1.10.0
• CPU Architecture:
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, libtorch, source): conda
• Build command you used (if compiling from source):
• Are you using local sources or building from archives:
• Python version: 3.8
• CUDA version: 11.1
• GPU models and configuration: 3090
• Any other relevant information:

Additional context

[p1x31]

Hi, try to rename forward method in TestEmbedding class and trace the model with torch.jit.trace_module

Also, are you loading weights or just compiling the graph?

[hturki]

Renaming forward to forward2 in TestEmbedding and using trace_module leads to the same errors as in the original description.

I'd ultimately like to be able to have this work after loading weights, but the repro in the description above happens when just compiling the graph (the code snippet should be fully self-contained if you want to give it a try)

[ignitemylife]

I have met the same error in different situation , I wonder there may be some ops/sub graph which trtorch not support well when the 'dynamic input shapes' set. So here comes the question: how can we solve it?

[p1x31]

Hi @hturki, I've given it a try and got this error. Could you check if both forward methods are correct

Traceback (most recent call last):
  File "nerf.py", line 75, in <module>
    traced_model_cuda = torch.jit.trace_module(sub_model, inputs)
  File "anaconda3/lib/python3.8/site-packages/torch/jit/_trace.py", line 952, in trace_module
    module._c._create_method_from_trace(
  File "anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1039, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "nerf.py", line 50, in forward
    input_xyz = self.embedding_xyz(x[:,:3])
  File "/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1039, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 201, in _forward_unimplemented
    raise NotImplementedError
NotImplementedError

upper bound and larger bound inconsistent with step sign means there are something wrong with indices e.g starting index larger than the beginning
in the line for i, freq in enumerate(self.freq_bands): iis never accessed

[github-actions]

This issue has not seen activity for 90 days, Remove stale label or comment or this will be closed in 10 days

[hturki]

Hi @p1x31 - sorry for the very delayed reply. I tried this again against Torch-TensorRT 1.0.0 and updated the code in the original bug description accordingly and am running into a different error message now:

scripts/tensorrt-test.py:13: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for freq in self.freq_bands:
before compile tensor([[0.0135, 0.0424, 0.0161, 0.0345]], grad_fn=<CatBackward0>)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::processCheck::581] Error Code 4: Internal Error (%257 : Tensor = aten::matmul(%input.1, %256): attempt to multiply two matrices with mismatching dimensions)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 2: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
Traceback (most recent call last):
  File "scripts/tensorrt-test.py", line 66, in <module>
    trt_ts_module = torch_tensorrt.compile(sub_model,
  File "/home/hturki/miniconda3/envs/nerf_slam/lib/python3.8/site-packages/torch_tensorrt/_compile.py", line 97, in compile
    return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs)
  File "/home/hturki/miniconda3/envs/nerf_slam/lib/python3.8/site-packages/torch_tensorrt/ts/_compiler.py", line 119, in compile
    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: [Error thrown at core/conversion/conversionctx/ConversionCtx.cpp:162] Building serialized network failed in TensorRT

Note that the model seems to work properly before compilation, as evidenced by the print statement.

[p1x31]

Hi @hturki - sorry for the late reply. Try passing the tensor you're originally tracing with:

input = torch.ones(1,3).to("cuda")
sub_model = torch.jit.trace(TestNeRF().to("cuda"), input)
print('before compile', sub_model(torch.ones(1, 3).to("cuda")))

trt_ts_module = torch_tensorrt.compile(sub_model.to("cuda"),
                                        inputs=[input#torch_tensorrt.Input(
                                        #    min_shape=[1, 3, 224, 224],
                                        #    opt_shape=[1, 3, 512, 512],
                                        #    max_shape=[1, 3, 1024, 1024],
                                           #shape=[1, 3, 256, 256],
                                        #   dtype=torch.float)
                                       ],
                                       enabled_precisions={torch.float}
                                       )

print('after compile', trt_ts_module(torch.ones(1, 3).to("cuda")))

py:13: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for freq in self.freq_bands:
before compile tensor([[-0.0127,  0.0096,  0.0209,  0.0551]], device='cuda:0',
       grad_fn=<CatBackward0>)
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Detected invalid timing cache, setup a local cache instead
after compile tensor([[-0.0127,  0.0096,  0.0209,  0.0551]], device='cuda:0')

Torch-TensorRT 1.1.0a0+b652045d

[hturki]

Thank you @p1x31 - I can confirm that my first snippet now works!

Now that that simple example works, I tried introducing an nn.Embedding into the model as per the below:

from typing import Optional, List

import torch
import torch_tensorrt
import torch.nn as nn
import torch.nn.functional as F


class TestEmbedding(nn.Module):
    def __init__(self, N_freqs):
        super(TestEmbedding, self).__init__()
        self.freq_bands = 2 ** torch.linspace(0, N_freqs - 1, N_freqs)

    def forward2(self, x):
        out = [x]
        for freq in self.freq_bands:
            out += [torch.sin(freq * x), torch.cos(freq * x)]

        return torch.cat(out, -1)

class TestNeRF(nn.Module):
    def __init__(self, pos_xyz_dim: int, pos_dir_dim: int, layers: int, skip_layers: List[int], layer_dim: int,
                 appearance_dim: int, appearance_count: int, rgb_dim: int, xyz_dim: int):
        super(TestNeRF, self).__init__()
        self.xyz_dim = xyz_dim
        self.embedding_xyz = TestEmbedding(pos_xyz_dim)
        in_channels_xyz = xyz_dim + xyz_dim * pos_xyz_dim * 2

        self.skip_layers = skip_layers

        xyz_encodings = []

        # xyz encoding layers
        for i in range(layers):
            if i == 0:
                layer = nn.Linear(in_channels_xyz, layer_dim)
            elif i in skip_layers:
                layer = nn.Linear(layer_dim + in_channels_xyz, layer_dim)
            else:
                layer = nn.Linear(layer_dim, layer_dim)
            layer = nn.Sequential(layer, nn.ReLU(True))
            xyz_encodings.append(layer)

        self.xyz_encodings = nn.ModuleList(xyz_encodings)

        if pos_dir_dim > 0:
            self.embedding_dir = TestEmbedding(pos_dir_dim)
            in_channels_dir = 3 + 3 * pos_dir_dim * 2
        else:
            self.embedding_dir = None
            in_channels_dir = 0

        if appearance_dim > 0:
            self.embedding_a = nn.Embedding(appearance_count, appearance_dim)
        else:
            self.embedding_a = None

        if pos_dir_dim > 0 or appearance_dim > 0:
            self.xyz_encoding_final = nn.Linear(layer_dim, layer_dim)
            # direction and appearance encoding layers
            self.dir_a_encoding = nn.Sequential(nn.Linear(layer_dim + in_channels_dir + appearance_dim, layer_dim // 2),
                                                nn.ReLU(True))
        else:
            self.xyz_encoding_final = None

        # output layers
        self.sigma = nn.Linear(layer_dim, 1)
        self.sigma_activation = nn.ReLU()

        rgb = nn.Linear(layer_dim // 2 if (pos_dir_dim > 0 or appearance_dim > 0) else layer_dim, rgb_dim)
        if rgb_dim == 3:
            self.rgb = nn.Sequential(rgb, nn.Sigmoid())
        else:
            self.rgb = rgb  # We're using spherical harmonics and will convert to sigmoid in rendering.py

    def forward(self, x: torch.Tensor, sigma_only: bool = False,
                sigma_noise: Optional[torch.Tensor] = None) -> torch.Tensor:
        input_xyz = self.embedding_xyz.forward2(x[:, :self.xyz_dim])
        xyz_ = input_xyz
        for i, xyz_encoding in enumerate(self.xyz_encodings):
            if i in self.skip_layers:
                xyz_ = torch.cat([input_xyz, xyz_], -1)
            xyz_ = xyz_encoding(xyz_)

        sigma = self.sigma(xyz_)
        if sigma_noise is not None:
            sigma += sigma_noise

        sigma = self.sigma_activation(sigma)

        if sigma_only:
            return sigma

        if self.xyz_encoding_final is not None:
            xyz_encoding_final = self.xyz_encoding_final(xyz_)
            dir_a_encoding_input = [xyz_encoding_final]

            if self.embedding_dir is not None:
                dir_a_encoding_input.append(self.embedding_dir.forward2(x[:, -4:-1]))

            if self.embedding_a is not None:
                dir_a_encoding_input.append(self.embedding_a(x[:, -1].long()))

            dir_a_encoding = self.dir_a_encoding(torch.cat(dir_a_encoding_input, -1))
            rgb = self.rgb(dir_a_encoding)
        else:
            rgb = self.rgb(xyz_)

        return torch.cat([rgb, sigma], -1)


ones = torch.ones(1, 7).cuda()
sub_model = torch.jit.trace(TestNeRF(10, 4, 8, [4], 256, 16, 10, 3, 3).cuda(), ones)
print('before compile', sub_model(torch.ones(1, 7).cuda()))

trt_ts_module = torch_tensorrt.compile(sub_model,
                                       inputs=[ones],
                                       enabled_precisions={torch.float}
                                       )

print('after compile', trt_ts_module(torch.ones(1, 7).cuda()))

I now get the following error:

scripts/tensorrt-test2.py:16: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for freq in self.freq_bands:
before compile tensor([[0.4709, 0.4592, 0.5017, 0.0183]], device='cuda:0',
       grad_fn=<CatBackward0>)
Traceback (most recent call last):
  File "scripts/tensorrt-test2.py", line 116, in <module>
    trt_ts_module = torch_tensorrt.compile(sub_model,
  File "/home/hturki/miniconda3/envs/nerf_slam/lib/python3.8/site-packages/torch_tensorrt/_compile.py", line 97, in compile
    return torch_tensorrt.ts.compile(ts_mod, inputs=inputs, enabled_precisions=enabled_precisions, **kwargs)
  File "/home/hturki/miniconda3/envs/nerf_slam/lib/python3.8/site-packages/torch_tensorrt/ts/_compiler.py", line 119, in compile
    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: [Error thrown at core/util/trt_util.cpp:283] Expected type to be true but got false
Unsupported ATen data type Long

Is this to be expected / are there plans in the near future to support long tensors?