failing engine building of llama-2 on V100 DGX

[abhinav-vimal13]

build command:
python build.py --model_dir ./hf-llama-2-7b/ \ --dtype float16 \ --remove_input_padding
--use_gpt_attention_plugin float16
--use_gemm_plugin float16
--output_dir ./hf-llama-2-7b/trt_engines/fp16/1-gpu/

during building, I am getting Device memory is insufficient to use tactic issue and process getting killed automatically, can you please tell me about this.

[11/16/2023-04:25:27] [TRT] [W] Tactic Device request: 66048MB Available: 32510MB. Device memory is insufficient to use tactic.
[11/16/2023-04:25:27] [TRT] [W] UNSUPPORTED_STATESkipping tactic 0 due to insufficient memory on requested size of 66048 detected for tactic 0x0000000000000018.
[11/16/2023-04:25:27] [TRT] [W] Tactic Device request: 66048MB Available: 32510MB. Device memory is insufficient to use tactic.
[11/16/2023-04:25:28] [TRT] [W] UNSUPPORTED_STATESkipping tactic 1 due to insufficient memory on requested size of 66048 detected for tactic 0x0000000000000019.
[11/16/2023-04:25:28] [TRT] [W] Tactic Device request: 66048MB Available: 32510MB. Device memory is insufficient to use tactic.
[11/16/2023-04:25:28] [TRT] [W] UNSUPPORTED_STATESkipping tactic 2 due to insufficient memory on requested size of 66048 detected for tactic 0x000000000000001a.
[11/16/2023-04:25:28] [TRT] [W] Tactic Device request: 66048MB Available: 32510MB. Device memory is insufficient to use tactic.
[11/16/2023-04:25:28] [TRT] [W] UNSUPPORTED_STATESkipping tactic 3 due to insufficient memory on requested size of 66048 detected for ta

[byshiue]

You can try enable --enable_context_fmha and use small batch_size/input_length/output_length to reduce the memory requirement first.

[abhinav-vimal13]

after enabling --enable_context_fmha, I was getting unsupported architecture issue pasting below

(venv) root@e8c9d86cc3cb:/workspace/TensorRT-LLM/examples/llama# python build.py --model_dir ./hf-llama-2-7b/ \ --dtype float16 \ --remove_input_padding
--use_gpt_attention_plugin float16
--enable_context_fmha
--use_gemm_plugin float16
--output_dir ./hf-llama-2-7b/trt_engines/fp16/1-gpu/
[11/10/2023-11:41:05] [TRT-LLM] [I] Serially build TensorRT engines.
[11/10/2023-11:41:05] [TRT] [I] [MemUsageChange] Init CUDA: CPU +14, GPU +0, now: CPU 119, GPU 5475 (MiB)
[11/10/2023-11:41:07] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +482, GPU +80, now: CPU 737, GPU 5555 (MiB)
[11/10/2023-11:41:07] [TRT-LLM] [W] Invalid timing cache, using freshly created one
[11/10/2023-11:41:15] [TRT-LLM] [I] Loading HF LLaMA ... from ./hf-llama-2-7b/
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████| 2/2 [00:14<00:00, 7.26s/it]
[11/10/2023-11:41:30] [TRT-LLM] [I] HF LLaMA loaded. Total time: 00:00:14
[11/10/2023-11:41:30] [TRT-LLM] [I] Loading weights from HF LLaMA...
[11/10/2023-11:41:33] [TRT-LLM] [I] Weights loaded. Total time: 00:00:02
[11/10/2023-11:41:34] [TRT-LLM] [I] Context FMHA Enabled
[11/10/2023-11:41:34] [TRT-LLM] [I] Remove Padding Enabled
[11/10/2023-11:41:34] [TRT] [W] A shape layer can only run in INT32 precision
[11/10/2023-11:41:34] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/vocab_embedding/GATHER_0_output_0 and LLaMAForCausalLM/layers/0/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/10/2023-11:41:34] [TRT] [W] A shape layer can only run in INT32 precision
[11/10/2023-11:41:34] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/0/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
terminate called after throwing an instance of 'tensorrt_llm::common::TllmException'
what(): [TensorRT-LLM][ERROR] Assertion failed: Unsupported architecture (/workspace/TensorRT-LLM/cpp/tensorrt_llm/kernels/contextFusedMultiHeadAttention/fmhaRunner.cpp:87)
1 0x7fb0b1200e0b /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(+0x34e0b) [0x7fb0b1200e0b]
2 0x7fb0b12a35a7 /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(+0xd75a7) [0x7fb0b12a35a7]
3 0x7fb0b125a2af tensorrt_llm::plugins::GPTAttentionPluginCommon::initialize() + 399
4 0x7fb0b127260c tensorrt_llm::plugins::GPTAttentionPlugin* tensorrt_llm::plugins::GPTAttentionPluginCommon::cloneImpl<tensorrt_llm::plugins::GPTAttentionPlugin>() const + 396
5 0x7fb271912599 /usr/local/tensorrt/lib/libnvinfer.so.9(+0xa87599) [0x7fb271912599]
6 0x7fb27187f63e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x9f463e) [0x7fb27187f63e]
7 0x7fb2808dc40f /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt/tensorrt.so(+0xdc40f) [0x7fb2808dc40f]
8 0x7fb280842ea3 /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt/tensorrt.so(+0x42ea3) [0x7fb280842ea3]
9 0x561081db799e python(+0x15c99e) [0x561081db799e]
10 0x561081dae4ab _PyObject_MakeTpCall + 603
11 0x561081dc5f0b python(+0x16af0b) [0x561081dc5f0b]
12 0x561081da6462 _PyEval_EvalFrameDefault + 25522
13 0x561081db81ec _PyFunction_Vectorcall + 124
14 0x561081dc68e2 PyObject_Call + 290
15 0x561081da2af0 _PyEval_EvalFrameDefault + 10816
16 0x561081db81ec _PyFunction_Vectorcall + 124
17 0x561081dc68e2 PyObject_Call + 290
18 0x561081da2af0 _PyEval_EvalFrameDefault + 10816
19 0x561081dc5c31 python(+0x16ac31) [0x561081dc5c31]
20 0x561081dc68e2 PyObject_Call + 290
21 0x561081da2af0 _PyEval_EvalFrameDefault + 10816
22 0x561081db81ec _PyFunction_Vectorcall + 124
23 0x561081dad6dd _PyObject_FastCallDictTstate + 365
24 0x561081dc2cac _PyObject_Call_Prepend + 92
25 0x561081ee0610 python(+0x285610) [0x561081ee0610]
26 0x561081dae4ab _PyObject_MakeTpCall + 603
27 0x561081da7683 _PyEval_EvalFrameDefault + 30163
28 0x561081dc5c31 python(+0x16ac31) [0x561081dc5c31]
29 0x561081dc68e2 PyObject_Call + 290
30 0x561081da2af0 _PyEval_EvalFrameDefault + 10816
31 0x561081db81ec _PyFunction_Vectorcall + 124
32 0x561081dad6dd _PyObject_FastCallDictTstate + 365
33 0x561081dc2cac _PyObject_Call_Prepend + 92
34 0x561081ee0610 python(+0x285610) [0x561081ee0610]
35 0x561081dae4ab _PyObject_MakeTpCall + 603
36 0x561081da7683 _PyEval_EvalFrameDefault + 30163
37 0x561081dc5c31 python(+0x16ac31) [0x561081dc5c31]
38 0x561081da6462 _PyEval_EvalFrameDefault + 25522
39 0x561081dc5d7e python(+0x16ad7e) [0x561081dc5d7e]
40 0x561081da2af0 _PyEval_EvalFrameDefault + 10816
41 0x561081dad634 _PyObject_FastCallDictTstate + 196
42 0x561081dc2d11 _PyObject_Call_Prepend + 193
43 0x561081ee0610 python(+0x285610) [0x561081ee0610]
44 0x561081dc687b PyObject_Call + 187
45 0x561081da2af0 _PyEval_EvalFrameDefault + 10816
46 0x561081db81ec _PyFunction_Vectorcall + 124
47 0x561081da0785 _PyEval_EvalFrameDefault + 1749
48 0x561081db81ec _PyFunction_Vectorcall + 124
49 0x561081da0785 _PyEval_EvalFrameDefault + 1749
50 0x561081d9ced6 python(+0x141ed6) [0x561081d9ced6]
51 0x561081e93366 PyEval_EvalCode + 134
52 0x561081ec0108 python(+0x265108) [0x561081ec0108]
53 0x561081eb8f5b python(+0x25df5b) [0x561081eb8f5b]
54 0x561081ebfe55 python(+0x264e55) [0x561081ebfe55]
55 0x561081ebf338 _PyRun_SimpleFileObject + 424
56 0x561081ebf033 _PyRun_AnyFileObject + 67
57 0x561081eb02de Py_RunMain + 702
58 0x561081e8632d Py_BytesMain + 45
59 0x7fb281238d90 /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb281238d90]
60 0x7fb281238e40 __libc_start_main + 128
61 0x561081e86225 _start + 37
[e8c9d86cc3cb:90099] *** Process received signal ***
[e8c9d86cc3cb:90099] Signal: Aborted (6)
[e8c9d86cc3cb:90099] Signal code: (-6)
[e8c9d86cc3cb:90099] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fb281251520]
[e8c9d86cc3cb:90099] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fb2812a5a7c]
[e8c9d86cc3cb:90099] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fb281251476]
[e8c9d86cc3cb:90099] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fb2812377f3]
[e8c9d86cc3cb:90099] [ 4] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa2bbe)[0x7fb26e1eebbe]
[e8c9d86cc3cb:90099] [ 5] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae24c)[0x7fb26e1fa24c]
[e8c9d86cc3cb:90099] [ 6] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xad229)[0x7fb26e1f9229]
[e8c9d86cc3cb:90099] [ 7] /lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x99)[0x7fb26e1f9999]
[e8c9d86cc3cb:90099] [ 8] /lib/x86_64-linux-gnu/libgcc_s.so.1(+0x16c64)[0x7fb280cb9c64]
[e8c9d86cc3cb:90099] [ 9] /lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x12d)[0x7fb280cba6bd]
[e8c9d86cc3cb:90099] [10] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(+0x3a14f)[0x7fb0b120614f]
[e8c9d86cc3cb:90099] [11] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(_ZN12tensorrt_llm7plugins24GPTAttentionPluginCommon10initializeEv+0x18f)[0x7fb0b125a2af]
[e8c9d86cc3cb:90099] [12] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(_ZNK12tensorrt_llm7plugins24GPTAttentionPluginCommon9cloneImplINS0_18GPTAttentionPluginEEEPT_v+0x18c)[0x7fb0b127260c]
[e8c9d86cc3cb:90099] [13] /usr/local/tensorrt/lib/libnvinfer.so.9(+0xa87599)[0x7fb271912599]
[e8c9d86cc3cb:90099] [14] /usr/local/tensorrt/lib/libnvinfer.so.9(+0x9f463e)[0x7fb27187f63e]
[e8c9d86cc3cb:90099] [15] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt/tensorrt.so(+0xdc40f)[0x7fb2808dc40f]
[e8c9d86cc3cb:90099] [16] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt/tensorrt.so(+0x42ea3)[0x7fb280842ea3]
[e8c9d86cc3cb:90099] [17] python(+0x15c99e)[0x561081db799e]
[e8c9d86cc3cb:90099] [18] python(_PyObject_MakeTpCall+0x25b)[0x561081dae4ab]
[e8c9d86cc3cb:90099] [19] python(+0x16af0b)[0x561081dc5f0b]
[e8c9d86cc3cb:90099] [20] python(_PyEval_EvalFrameDefault+0x63b2)[0x561081da6462]
[e8c9d86cc3cb:90099] [21] python(_PyFunction_Vectorcall+0x7c)[0x561081db81ec]
[e8c9d86cc3cb:90099] [22] python(PyObject_Call+0x122)[0x561081dc68e2]
[e8c9d86cc3cb:90099] [23] python(_PyEval_EvalFrameDefault+0x2a40)[0x561081da2af0]
[e8c9d86cc3cb:90099] [24] python(_PyFunction_Vectorcall+0x7c)[0x561081db81ec]
[e8c9d86cc3cb:90099] [25] python(PyObject_Call+0x122)[0x561081dc68e2]
[e8c9d86cc3cb:90099] [26] python(_PyEval_EvalFrameDefault+0x2a40)[0x561081da2af0]
[e8c9d86cc3cb:90099] [27] python(+0x16ac31)[0x561081dc5c31]
[e8c9d86cc3cb:90099] [28] python(PyObject_Call+0x122)[0x561081dc68e2]
[e8c9d86cc3cb:90099] [29] python(_PyEval_EvalFrameDefault+0x2a40)[0x561081da2af0]
[e8c9d86cc3cb:90099] *** End of error message ***
Aborted (core dumped)
(venv) root@e8c9d86cc3cb:/workspace/TensorRT-LLM/examples/llama#

[nv-guomingz]

after enabling --enable_context_fmha, I was getting unsupported architecture issue pasting below

(venv) root@e8c9d86cc3cb:/workspace/TensorRT-LLM/examples/llama# python build.py --model_dir ./hf-llama-2-7b/ \ --dtype float16 \ --remove_input_padding --use_gpt_attention_plugin float16 --enable_context_fmha --use_gemm_plugin float16 --output_dir ./hf-llama-2-7b/trt_engines/fp16/1-gpu/ [11/10/2023-11:41:05] [TRT-LLM] [I] Serially build TensorRT engines. [11/10/2023-11:41:05] [TRT] [I] [MemUsageChange] Init CUDA: CPU +14, GPU +0, now: CPU 119, GPU 5475 (MiB) [11/10/2023-11:41:07] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +482, GPU +80, now: CPU 737, GPU 5555 (MiB) [11/10/2023-11:41:07] [TRT-LLM] [W] Invalid timing cache, using freshly created one [11/10/2023-11:41:15] [TRT-LLM] [I] Loading HF LLaMA ... from ./hf-llama-2-7b/ Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████| 2/2 [00:14<00:00, 7.26s/it] [11/10/2023-11:41:30] [TRT-LLM] [I] HF LLaMA loaded. Total time: 00:00:14 [11/10/2023-11:41:30] [TRT-LLM] [I] Loading weights from HF LLaMA... [11/10/2023-11:41:33] [TRT-LLM] [I] Weights loaded. Total time: 00:00:02 [11/10/2023-11:41:34] [TRT-LLM] [I] Context FMHA Enabled [11/10/2023-11:41:34] [TRT-LLM] [I] Remove Padding Enabled [11/10/2023-11:41:34] [TRT] [W] A shape layer can only run in INT32 precision [11/10/2023-11:41:34] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/vocab_embedding/GATHER_0_output_0 and LLaMAForCausalLM/layers/0/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float. [11/10/2023-11:41:34] [TRT] [W] A shape layer can only run in INT32 precision [11/10/2023-11:41:34] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/0/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float. terminate called after throwing an instance of 'tensorrt_llm::common::TllmException' what(): [TensorRT-LLM][ERROR] Assertion failed: Unsupported architecture (/workspace/TensorRT-LLM/cpp/tensorrt_llm/kernels/contextFusedMultiHeadAttention/fmhaRunner.cpp:87) 1 0x7fb0b1200e0b /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(+0x34e0b) [0x7fb0b1200e0b] 2 0x7fb0b12a35a7 /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(+0xd75a7) [0x7fb0b12a35a7] 3 0x7fb0b125a2af tensorrt_llm::plugins::GPTAttentionPluginCommon::initialize() + 399 4 0x7fb0b127260c tensorrt_llm::plugins::GPTAttentionPlugin* tensorrt_llm::plugins::GPTAttentionPluginCommon::cloneImpl<tensorrt_llm::plugins::GPTAttentionPlugin>() const + 396 5 0x7fb271912599 /usr/local/tensorrt/lib/libnvinfer.so.9(+0xa87599) [0x7fb271912599] 6 0x7fb27187f63e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x9f463e) [0x7fb27187f63e] 7 0x7fb2808dc40f /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt/tensorrt.so(+0xdc40f) [0x7fb2808dc40f] 8 0x7fb280842ea3 /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt/tensorrt.so(+0x42ea3) [0x7fb280842ea3] 9 0x561081db799e python(+0x15c99e) [0x561081db799e] 10 0x561081dae4ab _PyObject_MakeTpCall + 603 11 0x561081dc5f0b python(+0x16af0b) [0x561081dc5f0b] 12 0x561081da6462 _PyEval_EvalFrameDefault + 25522 13 0x561081db81ec _PyFunction_Vectorcall + 124 14 0x561081dc68e2 PyObject_Call + 290 15 0x561081da2af0 _PyEval_EvalFrameDefault + 10816 16 0x561081db81ec _PyFunction_Vectorcall + 124 17 0x561081dc68e2 PyObject_Call + 290 18 0x561081da2af0 _PyEval_EvalFrameDefault + 10816 19 0x561081dc5c31 python(+0x16ac31) [0x561081dc5c31] 20 0x561081dc68e2 PyObject_Call + 290 21 0x561081da2af0 _PyEval_EvalFrameDefault + 10816 22 0x561081db81ec _PyFunction_Vectorcall + 124 23 0x561081dad6dd _PyObject_FastCallDictTstate + 365 24 0x561081dc2cac _PyObject_Call_Prepend + 92 25 0x561081ee0610 python(+0x285610) [0x561081ee0610] 26 0x561081dae4ab _PyObject_MakeTpCall + 603 27 0x561081da7683 _PyEval_EvalFrameDefault + 30163 28 0x561081dc5c31 python(+0x16ac31) [0x561081dc5c31] 29 0x561081dc68e2 PyObject_Call + 290 30 0x561081da2af0 _PyEval_EvalFrameDefault + 10816 31 0x561081db81ec _PyFunction_Vectorcall + 124 32 0x561081dad6dd _PyObject_FastCallDictTstate + 365 33 0x561081dc2cac _PyObject_Call_Prepend + 92 34 0x561081ee0610 python(+0x285610) [0x561081ee0610] 35 0x561081dae4ab _PyObject_MakeTpCall + 603 36 0x561081da7683 _PyEval_EvalFrameDefault + 30163 37 0x561081dc5c31 python(+0x16ac31) [0x561081dc5c31] 38 0x561081da6462 _PyEval_EvalFrameDefault + 25522 39 0x561081dc5d7e python(+0x16ad7e) [0x561081dc5d7e] 40 0x561081da2af0 _PyEval_EvalFrameDefault + 10816 41 0x561081dad634 _PyObject_FastCallDictTstate + 196 42 0x561081dc2d11 _PyObject_Call_Prepend + 193 43 0x561081ee0610 python(+0x285610) [0x561081ee0610] 44 0x561081dc687b PyObject_Call + 187 45 0x561081da2af0 _PyEval_EvalFrameDefault + 10816 46 0x561081db81ec _PyFunction_Vectorcall + 124 47 0x561081da0785 _PyEval_EvalFrameDefault + 1749 48 0x561081db81ec _PyFunction_Vectorcall + 124 49 0x561081da0785 _PyEval_EvalFrameDefault + 1749 50 0x561081d9ced6 python(+0x141ed6) [0x561081d9ced6] 51 0x561081e93366 PyEval_EvalCode + 134 52 0x561081ec0108 python(+0x265108) [0x561081ec0108] 53 0x561081eb8f5b python(+0x25df5b) [0x561081eb8f5b] 54 0x561081ebfe55 python(+0x264e55) [0x561081ebfe55] 55 0x561081ebf338 _PyRun_SimpleFileObject + 424 56 0x561081ebf033 _PyRun_AnyFileObject + 67 57 0x561081eb02de Py_RunMain + 702 58 0x561081e8632d Py_BytesMain + 45 59 0x7fb281238d90 /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb281238d90] 60 0x7fb281238e40 __libc_start_main + 128 61 0x561081e86225 _start + 37 [e8c9d86cc3cb:90099] *** Process received signal *** [e8c9d86cc3cb:90099] Signal: Aborted (6) [e8c9d86cc3cb:90099] Signal code: (-6) [e8c9d86cc3cb:90099] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fb281251520] [e8c9d86cc3cb:90099] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fb2812a5a7c] [e8c9d86cc3cb:90099] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fb281251476] [e8c9d86cc3cb:90099] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fb2812377f3] [e8c9d86cc3cb:90099] [ 4] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa2bbe)[0x7fb26e1eebbe] [e8c9d86cc3cb:90099] [ 5] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae24c)[0x7fb26e1fa24c] [e8c9d86cc3cb:90099] [ 6] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xad229)[0x7fb26e1f9229] [e8c9d86cc3cb:90099] [ 7] /lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x99)[0x7fb26e1f9999] [e8c9d86cc3cb:90099] [ 8] /lib/x86_64-linux-gnu/libgcc_s.so.1(+0x16c64)[0x7fb280cb9c64] [e8c9d86cc3cb:90099] [ 9] /lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x12d)[0x7fb280cba6bd] [e8c9d86cc3cb:90099] [10] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(+0x3a14f)[0x7fb0b120614f] [e8c9d86cc3cb:90099] [11] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(_ZN12tensorrt_llm7plugins24GPTAttentionPluginCommon10initializeEv+0x18f)[0x7fb0b125a2af] [e8c9d86cc3cb:90099] [12] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt_llm/libs/libnvinfer_plugin_tensorrt_llm.so(_ZNK12tensorrt_llm7plugins24GPTAttentionPluginCommon9cloneImplINS0_18GPTAttentionPluginEEEPT_v+0x18c)[0x7fb0b127260c] [e8c9d86cc3cb:90099] [13] /usr/local/tensorrt/lib/libnvinfer.so.9(+0xa87599)[0x7fb271912599] [e8c9d86cc3cb:90099] [14] /usr/local/tensorrt/lib/libnvinfer.so.9(+0x9f463e)[0x7fb27187f63e] [e8c9d86cc3cb:90099] [15] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt/tensorrt.so(+0xdc40f)[0x7fb2808dc40f] [e8c9d86cc3cb:90099] [16] /workspace/TensorRT-LLM/venv/lib/python3.10/site-packages/tensorrt/tensorrt.so(+0x42ea3)[0x7fb280842ea3] [e8c9d86cc3cb:90099] [17] python(+0x15c99e)[0x561081db799e] [e8c9d86cc3cb:90099] [18] python(_PyObject_MakeTpCall+0x25b)[0x561081dae4ab] [e8c9d86cc3cb:90099] [19] python(+0x16af0b)[0x561081dc5f0b] [e8c9d86cc3cb:90099] [20] python(_PyEval_EvalFrameDefault+0x63b2)[0x561081da6462] [e8c9d86cc3cb:90099] [21] python(_PyFunction_Vectorcall+0x7c)[0x561081db81ec] [e8c9d86cc3cb:90099] [22] python(PyObject_Call+0x122)[0x561081dc68e2] [e8c9d86cc3cb:90099] [23] python(_PyEval_EvalFrameDefault+0x2a40)[0x561081da2af0] [e8c9d86cc3cb:90099] [24] python(_PyFunction_Vectorcall+0x7c)[0x561081db81ec] [e8c9d86cc3cb:90099] [25] python(PyObject_Call+0x122)[0x561081dc68e2] [e8c9d86cc3cb:90099] [26] python(_PyEval_EvalFrameDefault+0x2a40)[0x561081da2af0] [e8c9d86cc3cb:90099] [27] python(+0x16ac31)[0x561081dc5c31] [e8c9d86cc3cb:90099] [28] python(PyObject_Call+0x122)[0x561081dc68e2] [e8c9d86cc3cb:90099] [29] python(_PyEval_EvalFrameDefault+0x2a40)[0x561081da2af0] [e8c9d86cc3cb:90099] *** End of error message *** Aborted (core dumped) (venv) root@e8c9d86cc3cb:/workspace/TensorRT-LLM/examples/llama#

Yes, enable_context_mha opt is not applicable on volta platform. You may try to build the engine with smaller bs as @byshiue suggested above.

[abhinav-vimal13]

sure let me try that

[abhinav-vimal13]

I tried by reducing batch/input_len/output_len as below
parser.add_argument('--max_batch_size', type=int, default=1) #8
parser.add_argument('--max_input_len', type=int, default=128) #2048
parser.add_argument('--max_output_len', type=int, default=64) #512

and building process is getting killed in between now pasting it below

(venv) root@e8c9d86cc3cb:/workspace/TensorRT-LLM/examples/llama# python build.py --model_dir ./hf-llama-2-7b/ --dtype float16 --remove_input_padding --use_gpt_attention_plugin float16 --use_gemm_plugin float16 --output_dir ./hf-llama-2-7b/trt_engines/fp16/1-gpu/
[11/20/2023-06:25:18] [TRT-LLM] [I] Serially build TensorRT engines.
[11/20/2023-06:25:19] [TRT] [I] [MemUsageChange] Init CUDA: CPU +14, GPU +0, now: CPU 119, GPU 5463 (MiB)
[11/20/2023-06:25:22] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +482, GPU +80, now: CPU 737, GPU 5543 (MiB)
[11/20/2023-06:25:22] [TRT-LLM] [W] Invalid timing cache, using freshly created one
[11/20/2023-06:25:37] [TRT-LLM] [I] Loading HF LLaMA ... from ./hf-llama-2-7b/
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:27<00:00, 13.92s/it]
[11/20/2023-06:26:06] [TRT-LLM] [I] HF LLaMA loaded. Total time: 00:00:28
[11/20/2023-06:26:06] [TRT-LLM] [I] Loading weights from HF LLaMA...
[11/20/2023-06:26:11] [TRT-LLM] [I] Weights loaded. Total time: 00:00:05
[11/20/2023-06:26:12] [TRT-LLM] [I] Remove Padding Enabled
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/vocab_embedding/GATHER_0_output_0 and LLaMAForCausalLM/layers/0/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/0/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/0/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/0/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/1/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/1/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/1/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/1/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/1/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/1/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/1/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/1/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/2/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/2/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/2/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/2/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/2/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/2/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/2/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/2/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/3/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/3/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/3/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/3/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/3/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/3/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/3/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/3/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/4/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/4/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/4/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/4/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/4/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/4/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/4/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/4/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/5/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/5/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/5/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/5/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/5/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/5/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/5/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/5/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/6/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/6/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/6/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/6/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/6/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/6/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/6/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/6/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/7/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/7/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/7/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/7/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/7/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/7/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/7/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/7/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/8/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/8/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/8/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/8/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/8/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/8/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/8/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/8/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/9/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/9/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/9/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/9/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/9/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/9/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/9/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/9/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/10/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/10/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/10/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/10/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/10/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/10/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/10/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/10/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/11/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/11/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/11/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/11/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/11/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/11/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/11/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/11/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/12/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/12/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/12/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/12/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/12/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/12/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/12/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/12/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/13/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/13/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/13/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/13/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/13/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/13/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/13/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/13/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/14/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/14/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/14/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/14/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/14/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/14/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/14/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/14/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/15/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/15/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/15/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/15/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/15/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/15/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/15/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/15/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/16/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/16/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/16/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/16/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/16/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/16/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/16/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/16/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/17/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/17/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/17/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/17/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/17/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/17/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/17/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/17/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/18/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/18/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/18/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/18/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/18/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/18/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/18/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/18/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/19/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/19/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/19/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/19/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/19/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/19/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/19/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/19/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/20/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/20/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/20/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/20/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/20/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/20/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/20/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/20/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/21/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/21/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/21/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/21/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/21/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/21/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/21/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/21/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/22/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/22/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/22/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/22/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/22/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/22/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/22/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/22/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/23/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/23/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/23/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/23/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/23/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/23/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/23/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/23/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/24/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/24/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/24/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/24/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/24/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/24/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/24/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/24/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/25/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/25/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/25/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/25/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/25/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/25/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/25/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/25/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/26/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/26/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/26/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/26/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/26/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/26/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/26/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/26/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/27/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/27/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/27/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/27/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/27/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/27/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/27/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/27/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/28/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/28/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/28/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/28/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/28/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/28/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/28/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/28/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/29/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/29/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/29/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/29/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/29/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/29/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/29/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/29/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/30/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/30/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/30/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/30/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/30/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:12] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:12] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/30/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/30/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:13] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:13] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/30/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/31/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:13] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:13] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/31/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/31/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:13] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:13] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/31/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/31/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:13] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:13] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/31/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/31/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:13] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:13] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/31/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/ln_f/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:13] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:26:13] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/ln_f/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/ln_f/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:26:13] [TRT-LLM] [I] Build TensorRT engine llama_float16_tp1_rank0.engine
[11/20/2023-06:26:13] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[11/20/2023-06:26:13] [TRT] [W] Unused Input: position_ids
[11/20/2023-06:26:13] [TRT] [W] Detected layernorm nodes in FP16.
[11/20/2023-06:26:13] [TRT] [W] Running layernorm after self-attention in FP16 may cause overflow. Exporting the model to the latest available ONNX opset (later than opset 17) to use the INormalizationLayer, or forcing layernorm layers to run in FP32 precision can help with preserving accuracy.
[11/20/2023-06:26:13] [TRT] [W] [RemoveDeadLayers] Input Tensor position_ids is unused or used only at compile-time, but is not being removed.
[11/20/2023-06:26:13] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 26527, GPU 6119 (MiB)
[11/20/2023-06:26:13] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 26529, GPU 6129 (MiB)
[11/20/2023-06:26:13] [TRT] [W] TensorRT was linked against cuDNN 8.9.4 but loaded cuDNN 8.9.1
[11/20/2023-06:26:13] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[11/20/2023-06:26:13] [TRT] [I] Global timing cache in use. Profiling results in this builder pass will be stored.
[11/20/2023-06:27:00] [TRT] [I] [GraphReduction] The approximate region cut reduction algorithm is called.
[11/20/2023-06:27:00] [TRT] [I] Detected 41 inputs and 33 output network tensors.
[11/20/2023-06:27:20] [TRT] [I] Total Host Persistent Memory: 59600
[11/20/2023-06:27:20] [TRT] [I] Total Device Persistent Memory: 0
[11/20/2023-06:27:20] [TRT] [I] Total Scratch Memory: 168824960
[11/20/2023-06:27:20] [TRT] [I] [BlockAssignment] Started assigning block shifts. This will take 554 steps to complete.
[11/20/2023-06:27:20] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 41.9557ms to assign 11 blocks to 554 nodes requiring 247471616 bytes.
[11/20/2023-06:27:20] [TRT] [I] Total Activation Memory: 247471616
[11/20/2023-06:27:21] [TRT] [I] Total Weights Memory: 13476832296
[11/20/2023-06:27:21] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 26845, GPU 18993 (MiB)
[11/20/2023-06:27:21] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 26845, GPU 19003 (MiB)
[11/20/2023-06:27:21] [TRT] [W] TensorRT was linked against cuDNN 8.9.4 but loaded cuDNN 8.9.1
[11/20/2023-06:27:21] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 500 MiB, GPU 12853 MiB
[11/20/2023-06:27:21] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +0, GPU +12853, now: CPU 0, GPU 12853 (MiB)
Killed
(venv) root@e8c9d86cc3cb:/workspace/TensorRT-LLM/examples/llama# python build.py --model_dir ./hf-llama-2-7b/ --dtype float16 --remove_input_padding --use_gpt_attention_plugin float16 --use_gemm_plugin float16 --output_dir ./hf-llama-2-7b/trt_engines/fp16/1-gpu/
[11/20/2023-06:29:48] [TRT-LLM] [I] Serially build TensorRT engines.
[11/20/2023-06:29:48] [TRT] [I] [MemUsageChange] Init CUDA: CPU +14, GPU +0, now: CPU 119, GPU 5463 (MiB)
[11/20/2023-06:29:56] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +482, GPU +80, now: CPU 737, GPU 5543 (MiB)
[11/20/2023-06:29:56] [TRT-LLM] [W] Invalid timing cache, using freshly created one
[11/20/2023-06:30:12] [TRT-LLM] [I] Loading HF LLaMA ... from ./hf-llama-2-7b/
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:28<00:00, 14.01s/it]
[11/20/2023-06:30:41] [TRT-LLM] [I] HF LLaMA loaded. Total time: 00:00:28
[11/20/2023-06:30:41] [TRT-LLM] [I] Loading weights from HF LLaMA...
[11/20/2023-06:30:46] [TRT-LLM] [I] Weights loaded. Total time: 00:00:05
[11/20/2023-06:30:47] [TRT-LLM] [I] Remove Padding Enabled
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/vocab_embedding/GATHER_0_output_0 and LLaMAForCausalLM/layers/0/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/0/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/0/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/0/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/0/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/1/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/1/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/1/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/1/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/1/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/1/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/1/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/1/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/2/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/2/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/2/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/2/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/2/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/2/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/2/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/2/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/3/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/3/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/3/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/3/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/3/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/3/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/3/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/3/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/4/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/4/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/4/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/4/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/4/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/4/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/4/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/4/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/5/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/5/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/5/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/5/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/5/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/5/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/5/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/5/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/6/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/6/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/6/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/6/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/6/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/6/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/6/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/6/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/7/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/7/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/7/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/7/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/7/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/7/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/7/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/7/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/8/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/8/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/8/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/8/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/8/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/8/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/8/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/8/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/9/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/9/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/9/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/9/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/9/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/9/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/9/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/9/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/10/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/10/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/10/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/10/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/10/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/10/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/10/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/10/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/11/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/11/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/11/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/11/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/11/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/11/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/11/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/11/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/12/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/12/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/12/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/12/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/12/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/12/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/12/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/12/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/13/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/13/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/13/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/13/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/13/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/13/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/13/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/13/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/14/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/14/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/14/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/14/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/14/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/14/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/14/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/14/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/15/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/15/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/15/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/15/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/15/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/15/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/15/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/15/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/16/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/16/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/16/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/16/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/16/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/16/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/16/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/16/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/17/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/17/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/17/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/17/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/17/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/17/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/17/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/17/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/18/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/18/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/18/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/18/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/18/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/18/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/18/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/18/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/19/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/19/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/19/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/19/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/19/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/19/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/19/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/19/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/20/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/20/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/20/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/20/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/20/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/20/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/20/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/20/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/21/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/21/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/21/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/21/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/21/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/21/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/21/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/21/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/22/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/22/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/22/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/22/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/22/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/22/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/22/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/22/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/23/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/23/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/23/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/23/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/23/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/23/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/23/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/23/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/24/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/24/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/24/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/24/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/24/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/24/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/24/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/24/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/25/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/25/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/25/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/25/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/25/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/25/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/25/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/25/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/26/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/26/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/26/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/26/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/26/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/26/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/26/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/26/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/27/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/27/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/27/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/27/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/27/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/27/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/27/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/27/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/28/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/28/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/28/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/28/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/28/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/28/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/28/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/28/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/29/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/29/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/29/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/29/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/29/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/29/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/29/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/29/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/30/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/30/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/30/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/30/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/30/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/30/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/30/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/30/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/layers/31/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/31/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/31/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/31/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/layers/31/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/31/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/layers/31/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/layers/31/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/ln_f/SHUFFLE_0_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:47] [TRT] [W] A shape layer can only run in INT32 precision
[11/20/2023-06:30:47] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/ln_f/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/ln_f/SHUFFLE_1_output_0: first input has type Half but second input has type Float.
[11/20/2023-06:30:48] [TRT-LLM] [I] Build TensorRT engine llama_float16_tp1_rank0.engine
[11/20/2023-06:30:48] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[11/20/2023-06:30:48] [TRT] [W] Unused Input: position_ids
[11/20/2023-06:30:48] [TRT] [W] Detected layernorm nodes in FP16.
[11/20/2023-06:30:48] [TRT] [W] Running layernorm after self-attention in FP16 may cause overflow. Exporting the model to the latest available ONNX opset (later than opset 17) to use the INormalizationLayer, or forcing layernorm layers to run in FP32 precision can help with preserving accuracy.
[11/20/2023-06:30:48] [TRT] [W] [RemoveDeadLayers] Input Tensor position_ids is unused or used only at compile-time, but is not being removed.
[11/20/2023-06:30:48] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 26527, GPU 6119 (MiB)
[11/20/2023-06:30:48] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 26528, GPU 6129 (MiB)
[11/20/2023-06:30:48] [TRT] [W] TensorRT was linked against cuDNN 8.9.4 but loaded cuDNN 8.9.1
[11/20/2023-06:30:48] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[11/20/2023-06:30:48] [TRT] [I] Global timing cache in use. Profiling results in this builder pass will be stored.
[11/20/2023-06:31:33] [TRT] [I] [GraphReduction] The approximate region cut reduction algorithm is called.
[11/20/2023-06:31:33] [TRT] [I] Detected 41 inputs and 33 output network tensors.
[11/20/2023-06:31:51] [TRT] [I] Total Host Persistent Memory: 59600
[11/20/2023-06:31:51] [TRT] [I] Total Device Persistent Memory: 0
[11/20/2023-06:31:51] [TRT] [I] Total Scratch Memory: 40927872
[11/20/2023-06:31:51] [TRT] [I] [BlockAssignment] Started assigning block shifts. This will take 554 steps to complete.
[11/20/2023-06:31:51] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 53.2159ms to assign 11 blocks to 554 nodes requiring 50761728 bytes.
[11/20/2023-06:31:51] [TRT] [I] Total Activation Memory: 50761728
[11/20/2023-06:31:52] [TRT] [I] Total Weights Memory: 13476832296
[11/20/2023-06:31:53] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 26845, GPU 18993 (MiB)
[11/20/2023-06:31:53] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 26845, GPU 19003 (MiB)
[11/20/2023-06:31:53] [TRT] [W] TensorRT was linked against cuDNN 8.9.4 but loaded cuDNN 8.9.1
[11/20/2023-06:31:53] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 500 MiB, GPU 12853 MiB
[11/20/2023-06:31:53] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +0, GPU +12853, now: CPU 0, GPU 12853 (MiB)
Killed
(venv) root@e8c9d86cc3cb:/workspace/TensorRT-LLM/examples/llama#

[nv-guomingz]

Could u plz try below cmd? And May I know your gpu memory size, 16GB or 32GB?
python build.py --model_dir ./hf-llama-2-7b/ --dtype float16 --remove_input_padding --use_gpt_attention_plugin float16 --use_gemm_plugin float16 --output_dir ./hf-llama-2-7b/trt_engines/fp16/1-gpu/ --max_batch_size=1 --max_input_len=128 --max_ouput_len=20

[abhinav-vimal13]

gpu memory size is 32GB and can i use --world_size 2 --tp_size 2 because I have 2 GPUs with each 32GB of memory

[abhinav-vimal13]

Could u plz try below cmd? And May I know your gpu memory size, 16GB or 32GB? python build.py --model_dir ./hf-llama-2-7b/ --dtype float16 --remove_input_padding --use_gpt_attention_plugin float16 --use_gemm_plugin float16 --output_dir ./hf-llama-2-7b/trt_engines/fp16/1-gpu/ --max_batch_size=1 --max_input_len=128 --max_ouput_len=20

still getting killed

[abhinav-vimal13]

gpu memory size is 32GB and can i use --world_size 2 --tp_size 2 because I have 2 GPUs with each 32GB of memory

tried with --world_size 2 --tp_size 2 but getting killed while Serializing the engine

[abhinav-vimal13]

can you please resolve this issue if possible

[Kevinddddddd]

Maybe, you need more CPU memory. What is your CPU memory size?

[nv-guomingz]

Could u plz try below cmd? And May I know your gpu memory size, 16GB or 32GB? python build.py --model_dir ./hf-llama-2-7b/ --dtype float16 --remove_input_padding --use_gpt_attention_plugin float16 --use_gemm_plugin float16 --output_dir ./hf-llama-2-7b/trt_engines/fp16/1-gpu/ --max_batch_size=1 --max_input_len=128 --max_output_len=20

still getting killed

I can build the engine succesfully with above command on DGX-V100(32GB).