H20 Using random weights to infer llama2-13B results in a divide-by-zero error.

[zxs789]

System Info

Device:H20
Driver:535.161.07
cuda-toolkit:12.2.0

python env:
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
tensorrt 10.0.1
tensorrt-cu12-bindings 10.0.1
tensorrt-cu12-libs 10.0.1
tensorrt-llm 0.11.0.dev2024052800

Who can help?

@ncomly-nvidia @kaiyux

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1.convert_config/llama_13b/float16/2-gpu/config.json
{ "architecture": "LlamaForCausalLM", "dtype": "float16", "logits_dtype": "float32", "vocab_size": 32000, "max_position_embeddings": 4096, "hidden_size": 5120, "num_hidden_layers": 40, "num_attention_heads": 40, "num_key_value_heads": 40, "head_size": 128, "hidden_act": "silu", "intermediate_size": 13824, "norm_epsilon": 1e-05, "position_embedding_type": "rope_gpt_neox", "use_parallel_embedding": false, "embedding_sharding_dim": 0, "share_embedding_table": false, "mapping": { "world_size": 2, "tp_size": 2, "pp_size": 1 }, "quantization": { "quant_algo": null, "kv_cache_quant_algo": null, "group_size": 128, "smoothquant_val": null, "has_zero_point": false, "pre_quant_scale": false, "exclude_modules": [ "lm_head" ] }, "kv_dtype": "float16", "rotary_scaling": null, "moe_normalization_mode": null, "rotary_base": 10000.0, "moe_num_experts": 0, "moe_top_k": 0, "moe_tp_mode": 2, "attn_bias": false, "disable_weight_only_quant_plugin": false, "mlp_bias": false }

2. run_build_llama13b.sh

#!/bin/bash

### 1. generate config json file

model=llama_13b
tp=2
dtype=fp16

model_config=./convert_config/$model/float16/${tp}-gpu/config.json
output_dir=./engines/$model/trt_engines/fp16/${tp}-gpu

### 2. generate engine with batches and input lens
max_output_len=200
declare -a input_lengths=(1024)

for ((i=0; i<${#input_lengths[@]}; i++)); do
  max_input_len=${input_lengths[$i]}

  case $max_input_len in
    1024) test_batch_sizes=(1 2 4 8 16 32 64 128 256) ;;
    *) echo "Invalid input length"; exit 1 ;;
  esac
  
  for b in "${test_batch_sizes[@]}"; do
    max_batch_size=$b
    echo "Running trtllm-build with max_input_len=$max_input_len, max_output_len=$max_output_len, max_batch_size=$max_batch_size"

    trtllm-build \
      --model_config $model_config \
      --gemm_plugin auto \
      --output_dir $output_dir

    if [ $? -ne 0 ]; then
      echo "trtllm-build failed for max_input_len=$max_input_len, max_batch_size=$max_batch_size"
    fi

    ### 3. begin test cases
    in=$max_input_len
    out=200

    work_dir=`pwd`
    engine_dir=./engines/$model/trt_engines/fp16/${tp}-gpu/

    echo "Running gptSessionBenchmark with input_len=$in, output_len=$out, batch_size=$b"
    mpirun --allow-run-as-root -n ${tp} python3 benchmarks/python/benchmark.py \
                                           --model ${model} \
                                           --mode plugin \
                                           --batch_size  "${b}" \
                                           --input_output_len "${in},${out}" \
                                           --warm_up 1 \
                                           --num_runs 4 \
                                           --engine_dir $engine_dir \
                                           --csv

    if [ $? -ne 0 ]; then
      echo "gptSessionBenchmark failed for input_len=$in, output_len=$out, batch_size=$b"
    fi
    ### 4. delete engine dir
    rm -rf $engine_dir
  done  
done
echo "All runs completed successfully."

Expected behavior

Expect to print performance data normally.

actual behavior

[96/03/2024-16:26:43] [TRT-LLM] [I] Engine serialized. Total time: 00:00:05
[06/03/2024-16:26:43] [TRT-LLM] [I] Total time of building all engines: 00:01:18
Running gptSessionBenchmark with input_len=1024,.output_len=200, batch_size=1
[TensorRT-LLM] TensorRT-LLM version: 0.11.0.dev2024052800
Allocated 117.50 MiB for execution context memory:
model_name,world_size,num_heads,num_kv_heads,num_layers,hidden_size,vocab_size,precision,batch_size,gpu_weights_percent,input_length,output_length,gpu_peak_mem(gb),build_time(s),tokens_per_sec,percentile95(ms),percentile99(ms),latency(ms),compute_cap,quantization,generation_time(ms),total_generated_tokens,generation_tokens_per_second
/root/anaconda3/envs/trt_llm/lib/python3.1o/site-packages/torch/nested/__init__py:166: UserWarning: The PyTorch API of nested tensors is
 in prototype stage and will change in the near future. (Triggered internally at . ./aten/src/ATen/NestedTensorImpl.cpp:178.)
return _nested.nested_tensor(
[dlp2-29-140:159817] *** Process received signal ***
[dlp2-29-140:159817] Signal: Floating point exception (8)
[dlp2-29-140:159817] Signal code: Integer divide-by-zero (1)
[dlp2-29-140:159817] Failing at address: 0x2b228fc4ec59
[dlp2-29-140:159817] [ 0] /lib64/libpthread.s0.0(+0xf630)[0x2b220e562630]
[dlp2-29-140:159817] [ 1] /root/anaconda3/envs/trt_llm/lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublasLt.so.12(+
0xa0bc59)[0x2b228fc4ec59]
[dlp2-29-140:159817]_[_21 /root/anaconda3/envs/trt_llm/lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublasLt.so.12(+
0x814383)[0x2b228fa57383]
[dlp2-29-140:159817]_[_3] /root/anaconda3/envs/trt_llm/lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublasLt.so.12(+
0x6ace72][0x2b228f8efe72]
[dlp2-29-140:159817] .[.4] /root/anaconda3/envs/trt_llm//lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublasLt.so.12(+
0x7aa087)[0x2b228f9ed087]
[dlp2-29-140:159817] [.5] /root/anaconda3/envs/trt_llm//lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublasLt.so.12(+
0x7ab055][0x2b228f9ee055]
[dlp2-29-140:159817] [ 6] /root/anaconda3/envs/trt_llm/lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublasLt.so.12(+
0x7ab774)[0x2b228f9ee774]
[dlp2-29-140:159817]. [_7]./root/anaconda3/envs/trt_llm/lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublasLt.so.12(c
ublasLtMatmul+0x1525)[0x2b228f9f2375]
[dlp2-29-140:159817] [ 8] /root/anaconda3/envs/trt_llm/lib/python3.10/site-packages/tensorrt_llm/libs/libtensorrt_llm.so(_zN12tensorrt_ll
m6common15CublasMMWrapper4GemmE17cublas0peration_tS2_iiiPKviS4_iPviffRK20cublasLtMatmuLAlgo_tbb+0xfd)[0x2b23b55e7a8d]
[dlp2-29-140:159817] [ 9] /root/anaconda3/envs/trt_llm/lib/python3.10/site-packages/tensorrt_llm/libs/libtensorrt_llm.so(_zN12tensorrt_ll
m6common15CublasMMWrapper4GemmE17cublas0peration_tS2_iiiPKviS4_iPviRKSt8optionalI31cublasLtMatmulHeuristicResult_tE+0x60)[0x2b23b55e7f70]

additional notes

The same command and script can run normally on the A100, but it will report an error of divide-by-zero on the H20. Is it because the NVIDIA CUDA version is too low?

[hijkzzz]

Does this happen under normal llama3 weights (from huggingface)?

[zxs789]

Does this happen under normal llama3 weights (from huggingface)?

Thanks, I will try normal llama weights.

[zxs789]

tensorrt_llm/parameter.py

def set_all_one_dummy(self):
        self.value = np.ones(self._shape, trt_dtype_to_np(self._dtype))

tensorrt_llm/models/modeling_utils.py

for name, param in model.named_parameters():
        param.set_all_one_dummy()

When I tried set all weights to one when build engine, get the same error of divide-by-zero

[zxs789]

Confirm one more thing, does TensorRT-LLM support H20 device?

[hijkzzz]

Confirm one more thing, does TensorRT-LLM support H20 device?

Thanks, we're trying to figure out why. Can you give us a specific command that went wrong? This includes converting and building commands.

I just tried the commands on H20, and it works well:

python examples/llama/convert_checkpoint.py --model_dir ./tmp/llama-v2-13b-hf/ --output_dir ./tmp/llama2_13b/ckpt
trtllm-build --checkpoint_dir ./tmp/llama2_13b/ckpt/ --gemm_plugin float16 --output_dir ./tmp/llama2_13b/engine

logs
[06/04/2024-09:36:55] [TRT] [I] Serialized 152374 bytes of compilation cache.
[06/04/2024-09:36:55] [TRT] [I] Serialized 8 timing cache entries
[06/04/2024-09:36:55] [TRT-LLM] [I] Timing cache serialized to model.cache
[06/04/2024-09:36:55] [TRT-LLM] [I] Serializing engine to ./tmp/llama2_13b/engine/rank0.engine...
[06/04/2024-09:38:05] [TRT-LLM] [I] Engine serialized. Total time: 00:01:10
[06/04/2024-09:38:06] [TRT-LLM] [I] Total time of building all engines: 00:04:13

[zxs789]

I tried chatglm2_6b_tp1 and llama2_13b_tp2 on H20 GPU, and both encountered a divide-by-zero issue. I used random weights, and when I changed all the weights to 1, this problem still occurred. Here are the steps to reproduce it:

System Info
Device:H20
Driver:535.161.07
cuda-toolkit:12.2.0

python env:
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
tensorrt 10.0.1
tensorrt-cu12-bindings 10.0.1
tensorrt-cu12-libs 10.0.1
tensorrt-llm 0.11.0.dev2024052800
torch 2.3.0

1. Create the config file:

convert_config/chatglm2_6b/float16/1-gpu/config.json

{
    "architecture": "ChatGLMForCausalLM",
    "dtype": "float16",
    "logits_dtype": "float32",
    "num_hidden_layers": 28,
    "num_attention_heads": 32,
    "num_key_value_heads": 2,
    "hidden_size": 4096,
    "intermediate_size": 13696,
    "norm_epsilon": 1e-05,
    "vocab_size": 65024,
    "position_embedding_type": "rope_gptj",
    "max_position_embeddings": 32768,
    "hidden_act": "swiglu",
    "use_parallel_embedding": false,
    "embedding_sharding_dim": 0,
    "share_embedding_table": false,
    "quantization": {
        "quant_algo": null,
        "kv_cache_quant_algo": null
    },
    "mapping": {
        "world_size": 1,
        "tp_size": 1,
        "pp_size": 1
    },
    "chatglm_version": "chatglm2",
    "add_bias_linear": false,
    "add_qkv_bias": true,
    "apply_query_key_layer_scaling": false,
    "apply_residual_connection_post_layernorm": false,
    "rmsnorm": true,
    "rope_ratio": 1.0
}

2. build the engine with model_config

build.sh

#!/bin/bash
model=chatglm2_6b
tp=1
dtype=fp16

model_config=./convert_config/$model/float16/${tp}-gpu/config.json
output_dir=./engines/$model/trt_engines/fp16/${tp}-gpu
max_batch_size=1
max_input_len=1024
max_output_len=200

trtllm-build \
      --model_config $model_config \
      --gpt_attention_plugin float16 \
      --remove_input_padding enable \
      --context_fmha enable \
      --gemm_plugin float16 \
      --max_batch_size $max_batch_size \
      --output_dir $output_dir \
      --context_fmha_fp32_acc enable \
      --enable_xqa enable \
      --max_input_len $max_input_len \
      --max_output_len $max_output_len \
      --multi_block_mode enable \
      --paged_kv_cache disable \
      --remove_input_padding enable \
      --strongly_typed \
      --use_custom_all_reduce enable \
      --use_fused_mlp \
      --workers $tp

3. run benchmark

python3 benchmarks/python/benchmark.py --model chatglm2_6b --mode plugin --batch_size 1 --input_output_len "1024,200" --warm_up 1 --num_runs 4 --engine_dir ./engines/chatglm2_6b/trt_engines/fp16/1-gpu/

4. Below is the code where I changed all the weights to 1:

diff --git a/tensorrt_llm/models/modeling_utils.py b/tensorrt_llm/models/modeling_utils.py
index 5ea7e04..d571434 100644
--- a/tensorrt_llm/models/modeling_utils.py
+++ b/tensorrt_llm/models/modeling_utils.py
@@ -1186,4 +1186,8 @@ def load_model(
                            from_pruned=is_checkpoint_pruned)
         model.load(weights, from_pruned=is_checkpoint_pruned)
 
+    for name, param in model.named_parameters():
+        param.set_all_one_dummy()
+        print(name, param.shape, param.print_value())
+
     return model
diff --git a/tensorrt_llm/parameter.py b/tensorrt_llm/parameter.py
index 42dc42b..2e92e63 100644
--- a/tensorrt_llm/parameter.py
+++ b/tensorrt_llm/parameter.py
@@ -140,6 +140,12 @@ class Parameter:
 
         self.value = v
 
+    def print_value(self):
+        return self._value
+
+    def set_all_one_dummy(self):
+        self.value = np.ones(self._shape, trt_dtype_to_np(self._dtype))
+
     def _get_weights(self) -> trt.Weights:
         if isinstance(self._value, Tensor):
             self._value.producer.__class__ = trt.IConstantLayer

Thanks again.

[zxs789]

Confirm one more thing, does TensorRT-LLM support H20 device?

Thanks, we're trying to figure out why. Can you give us a specific command that went wrong? This includes converting and building commands.

I just tried the commands on H20, and it works well:

python examples/llama/convert_checkpoint.py --model_dir ./tmp/llama-v2-13b-hf/ --output_dir ./tmp/llama2_13b/ckpt
trtllm-build --checkpoint_dir ./tmp/llama2_13b/ckpt/ --gemm_plugin float16 --output_dir ./tmp/llama2_13b/engine

logs
[06/04/2024-09:36:55] [TRT] [I] Serialized 152374 bytes of compilation cache.
[06/04/2024-09:36:55] [TRT] [I] Serialized 8 timing cache entries
[06/04/2024-09:36:55] [TRT-LLM] [I] Timing cache serialized to model.cache
[06/04/2024-09:36:55] [TRT-LLM] [I] Serializing engine to ./tmp/llama2_13b/engine/rank0.engine...
[06/04/2024-09:38:05] [TRT-LLM] [I] Engine serialized. Total time: 00:01:10
[06/04/2024-09:38:06] [TRT-LLM] [I] Total time of building all engines: 00:04:13

Yes,I can build engine success, but when i run benchmarks/python/benchmark.py encountered an error

[hijkzzz]

Confirm one more thing, does TensorRT-LLM support H20 device?

Thanks, we're trying to figure out why. Can you give us a specific command that went wrong? This includes converting and building commands.
I just tried the commands on H20, and it works well:

python examples/llama/convert_checkpoint.py --model_dir ./tmp/llama-v2-13b-hf/ --output_dir ./tmp/llama2_13b/ckpt
trtllm-build --checkpoint_dir ./tmp/llama2_13b/ckpt/ --gemm_plugin float16 --output_dir ./tmp/llama2_13b/engine

logs
[06/04/2024-09:36:55] [TRT] [I] Serialized 152374 bytes of compilation cache.
[06/04/2024-09:36:55] [TRT] [I] Serialized 8 timing cache entries
[06/04/2024-09:36:55] [TRT-LLM] [I] Timing cache serialized to model.cache
[06/04/2024-09:36:55] [TRT-LLM] [I] Serializing engine to ./tmp/llama2_13b/engine/rank0.engine...
[06/04/2024-09:38:05] [TRT-LLM] [I] Engine serialized. Total time: 00:01:10
[06/04/2024-09:38:06] [TRT-LLM] [I] Total time of building all engines: 00:04:13

Yes,I can build engine success, but when i run benchmarks/python/benchmark.py encountered an error

Have you tried the weights/ckpt from HuggingFace? I remember that I ran the benchmark.py well with Chatglm3 yesterday.

[zxs789]

Confirm one more thing, does TensorRT-LLM support H20 device?

Thanks, we're trying to figure out why. Can you give us a specific command that went wrong? This includes converting and building commands.
I just tried the commands on H20, and it works well:

python examples/llama/convert_checkpoint.py --model_dir ./tmp/llama-v2-13b-hf/ --output_dir ./tmp/llama2_13b/ckpt
trtllm-build --checkpoint_dir ./tmp/llama2_13b/ckpt/ --gemm_plugin float16 --output_dir ./tmp/llama2_13b/engine

logs
[06/04/2024-09:36:55] [TRT] [I] Serialized 152374 bytes of compilation cache.
[06/04/2024-09:36:55] [TRT] [I] Serialized 8 timing cache entries
[06/04/2024-09:36:55] [TRT-LLM] [I] Timing cache serialized to model.cache
[06/04/2024-09:36:55] [TRT-LLM] [I] Serializing engine to ./tmp/llama2_13b/engine/rank0.engine...
[06/04/2024-09:38:05] [TRT-LLM] [I] Engine serialized. Total time: 00:01:10
[06/04/2024-09:38:06] [TRT-LLM] [I] Total time of building all engines: 00:04:13

Yes,I can build engine success, but when i run benchmarks/python/benchmark.py encountered an error

Have you tried the weights/ckpt from HuggingFace? I remember that I ran the benchmark.py well with Chatglm3 yesterday.

Not yet, I am still downloading the real weights from hugging face, can you reproduce this error using random weights

[hijkzzz]

Confirm one more thing, does TensorRT-LLM support H20 device?

Thanks, we're trying to figure out why. Can you give us a specific command that went wrong? This includes converting and building commands.
I just tried the commands on H20, and it works well:

python examples/llama/convert_checkpoint.py --model_dir ./tmp/llama-v2-13b-hf/ --output_dir ./tmp/llama2_13b/ckpt
trtllm-build --checkpoint_dir ./tmp/llama2_13b/ckpt/ --gemm_plugin float16 --output_dir ./tmp/llama2_13b/engine

logs
[06/04/2024-09:36:55] [TRT] [I] Serialized 152374 bytes of compilation cache.
[06/04/2024-09:36:55] [TRT] [I] Serialized 8 timing cache entries
[06/04/2024-09:36:55] [TRT-LLM] [I] Timing cache serialized to model.cache
[06/04/2024-09:36:55] [TRT-LLM] [I] Serializing engine to ./tmp/llama2_13b/engine/rank0.engine...
[06/04/2024-09:38:05] [TRT-LLM] [I] Engine serialized. Total time: 00:01:10
[06/04/2024-09:38:06] [TRT-LLM] [I] Total time of building all engines: 00:04:13

Yes,I can build engine success, but when i run benchmarks/python/benchmark.py encountered an error

Have you tried the weights/ckpt from HuggingFace? I remember that I ran the benchmark.py well with Chatglm3 yesterday.

Not yet, I am still downloading the real weights from hugging face, can you reproduce this error using random weights

No, I can't.
My commands

python examples/llama/convert_checkpoint.py --model_dir ./tmp/llama-v2-13b-hf/ --output_dir ./tmp/llama2_13b

trtllm-build --model_config ./tmp/llama2_13b/config.json --gemm_plugin float16 --output_dir ./tmp/llama2_13b/engine2 --gpt_attention_plugin float16 --remove_input_padding enable --context_fmha enable --gemm_plugin float16 --max_batch_size 1 --context_fmha_fp32_acc enable --enable_xqa enable --max_input_len 1024 --max_output_len 200 --multi_block_mode enable --paged_kv_cache disable --remove_input_padding enable --use_custom_all_reduce enable --use_fused_mlp

 python3 benchmarks/python/benchmark.py --model llama_13b --mode plugin --input_output_len "1024,200" --warm_up 1  --num_runs 4 --engine_dir ./tmp/llama2_13b/engine2/ --csv --batch_size 1

logs

Allocated 126.00 MiB for execution context memory.
model_name,world_size,num_heads,num_kv_heads,num_layers,hidden_size,vocab_size,precision,batch_size,gpu_weights_percent,input_length,output_length,gpu_peak_mem(gb),build_time(s),tokens_per_sec,percentile95(ms),percentile99(ms),latency(ms),compute_cap,quantization,generation_time(ms),total_generated_tokens,generation_tokens_per_second
/usr/local/lib/python3.10/dist-packages/torch/nested/__init__.py:166: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/NestedTensorImpl.cpp:178.)
  return _nested.nested_tensor(
[06/04/2024-10:05:30] [TRT-LLM] [W] Logger level already set from environment. Discard new verbosity: error
[TensorRT-LLM] TensorRT-LLM version: 0.11.0.dev2024060400


llama_13b,1,40,40,40,5120,32000,float16,1,1.0,1024,200,25.991,0,45.73,4403.05,4403.05,4373.095,sm90,QuantMode.0,4162.918,199.0,47.803

notice that my CUDA env is

NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4
trtllm 24.04 docker container

It is also recommended to update the TRT-LLM version to the latest (it will be updated to main branch today or tomorrow).

[zxs789]

I tried chatglm2_6b_tp1 and llama2_13b_tp2 on H20 GPU, and both encountered a divide-by-zero issue. I used random weights, and when I changed all the weights to 1, this problem still occurred. Here are the steps to reproduce it:

System Info Device:H20 Driver:535.161.07 cuda-toolkit:12.2.0

python env: nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 tensorrt 10.0.1 tensorrt-cu12-bindings 10.0.1 tensorrt-cu12-libs 10.0.1 tensorrt-llm 0.11.0.dev2024052800 torch 2.3.0

1. Create the config file:

convert_config/chatglm2_6b/float16/1-gpu/config.json

{
    "architecture": "ChatGLMForCausalLM",
    "dtype": "float16",
    "logits_dtype": "float32",
    "num_hidden_layers": 28,
    "num_attention_heads": 32,
    "num_key_value_heads": 2,
    "hidden_size": 4096,
    "intermediate_size": 13696,
    "norm_epsilon": 1e-05,
    "vocab_size": 65024,
    "position_embedding_type": "rope_gptj",
    "max_position_embeddings": 32768,
    "hidden_act": "swiglu",
    "use_parallel_embedding": false,
    "embedding_sharding_dim": 0,
    "share_embedding_table": false,
    "quantization": {
        "quant_algo": null,
        "kv_cache_quant_algo": null
    },
    "mapping": {
        "world_size": 1,
        "tp_size": 1,
        "pp_size": 1
    },
    "chatglm_version": "chatglm2",
    "add_bias_linear": false,
    "add_qkv_bias": true,
    "apply_query_key_layer_scaling": false,
    "apply_residual_connection_post_layernorm": false,
    "rmsnorm": true,
    "rope_ratio": 1.0
}

2. build the engine with model_config

build.sh

#!/bin/bash
model=chatglm2_6b
tp=1
dtype=fp16

model_config=./convert_config/$model/float16/${tp}-gpu/config.json
output_dir=./engines/$model/trt_engines/fp16/${tp}-gpu
max_batch_size=1
max_input_len=1024
max_output_len=200

trtllm-build \
      --model_config $model_config \
      --gpt_attention_plugin float16 \
      --remove_input_padding enable \
      --context_fmha enable \
      --gemm_plugin float16 \
      --max_batch_size $max_batch_size \
      --output_dir $output_dir \
      --context_fmha_fp32_acc enable \
      --enable_xqa enable \
      --max_input_len $max_input_len \
      --max_output_len $max_output_len \
      --multi_block_mode enable \
      --paged_kv_cache disable \
      --remove_input_padding enable \
      --strongly_typed \
      --use_custom_all_reduce enable \
      --use_fused_mlp \
      --workers $tp

3. run benchmark

python3 benchmarks/python/benchmark.py --model chatglm2_6b --mode plugin --batch_size 1 --input_output_len "1024,200" --warm_up 1 --num_runs 4 --engine_dir ./engines/chatglm2_6b/trt_engines/fp16/1-gpu/

4. Below is the code where I changed all the weights to 1:

diff --git a/tensorrt_llm/models/modeling_utils.py b/tensorrt_llm/models/modeling_utils.py
index 5ea7e04..d571434 100644
--- a/tensorrt_llm/models/modeling_utils.py
+++ b/tensorrt_llm/models/modeling_utils.py
@@ -1186,4 +1186,8 @@ def load_model(
                            from_pruned=is_checkpoint_pruned)
         model.load(weights, from_pruned=is_checkpoint_pruned)
 
+    for name, param in model.named_parameters():
+        param.set_all_one_dummy()
+        print(name, param.shape, param.print_value())
+
     return model
diff --git a/tensorrt_llm/parameter.py b/tensorrt_llm/parameter.py
index 42dc42b..2e92e63 100644
--- a/tensorrt_llm/parameter.py
+++ b/tensorrt_llm/parameter.py
@@ -140,6 +140,12 @@ class Parameter:
 
         self.value = v
 
+    def print_value(self):
+        return self._value
+
+    def set_all_one_dummy(self):
+        self.value = np.ones(self._shape, trt_dtype_to_np(self._dtype))
+
     def _get_weights(self) -> trt.Weights:
         if isinstance(self._value, Tensor):
             self._value.producer.__class__ = trt.IConstantLayer

Thanks again.

this is my commends with random weights,can you run this commends on h20,if it worked,i can update my driver and cuda version,and the tensorrt-llm version

[hijkzzz]

Hi, it also works well with the random weights + chatglm2_6b.
It may be an issue of Drvier and CUDA versions.

logs

trtllm-build --model_config ./tmp/chatglm2_6b/ckpt/config.json --gemm_plugin float16 --output_dir ./tmp/chatglm2_6b/engine2 --gpt_attention_plugin float16 --remove_input_padding enable --context_fmha enable --gemm_plugin float16 --max_batch_size 1 --context_fmha_fp32_acc enable --enable_xqa enable --max_input_len 1024 --max_output_len 200 --multi_block_mode enable --paged_kv_cache disable --remove_input_padding enable --use_custom_all_reduce enable --use_fused_mlp

python3 benchmarks/python/benchmark.py --model chatglm2_6b --mode plugin --input_output_len "1024,200" --warm_up 1  --num_runs 4 --engine_dir ./tmp/chatglm2_6b/engine2/ --csv --batch_size 1

...
Allocated 117.50 MiB for execution context memory.
model_name,world_size,num_heads,num_kv_heads,num_layers,hidden_size,vocab_size,precision,batch_size,gpu_weights_percent,input_length,output_length,gpu_peak_mem(gb),build_time(s),tokens_per_sec,percentile95(ms),percentile99(ms),latency(ms),compute_cap,quantization,generation_time(ms),total_generated_tokens,generation_tokens_per_second
/usr/local/lib/python3.10/dist-packages/torch/nested/__init__.py:166: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/NestedTensorImpl.cpp:178.)
  return _nested.nested_tensor(
[06/04/2024-10:28:33] [TRT-LLM] [W] Logger level already set from environment. Discard new verbosity: error
[TensorRT-LLM] TensorRT-LLM version: 0.11.0.dev2024060400
chatglm2_6b,1,32,2,28,4096,65024,float16,1,1.0,1024,200,12.433,0,105.65,1904.281,1908.865,1893.025,sm90,QuantMode.0,1796.244,199.0,110.78