[Bug]: GLM-4.1V lora trained model reports target_module mismatch error

[miridih-jyjang]

Your current environment

This model I finetuned using LLaMA-Factory using GLM-4.1V with Lora. The LoRA config is,

model

model_name_or_path: zai-org/GLM-4.1V-9B-Thinking #/data/shared/checkpoints/hugging_face/GLM-4.1V-9B-Thinking
#model_name_or_path: saves/glm4_1_v_9b/lora/version020/checkpoint-2000
image_max_pixels: 262144
video_max_pixels: 16384
trust_remote_code: true

method

stage: sft
do_train: true
finetuning_type: lora
lora_rank: 32
lora_target: all
deepspeed: examples/deepspeed/ds_z3_offload_config.json # choices: [ds_z0_config.json, ds_z2_config.json, ds_z3_config.json]

dataset

dataset: mllm_dataset #,identity,alpaca_en_demo # video: mllm_video_demo
template: glm4v
cutoff_len: 15000
max_samples: 1000000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

output

output_dir: saves/glm4_1_v_9b/lora/version020
logging_steps: 10
save_steps: 2000
plot_loss: true
overwrite_output_dir: true
save_only_model: false
save_strategy: steps #
metric_for_best_model: loss #
#save_total_limit: 4
report_to: wandb
run_name: glm4_1_v_9b_lora_sft_v020
load_best_model_at_end: true #

train

per_device_train_batch_size: 2
gradient_accumulation_steps: 1
learning_rate: 1.0e-4
num_train_epochs: 4.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

eval

eval_dataset: mllm_eval_dataset
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 100
do_predict: false
predict_with_generate: false

The output of DISABLE_VERSION_CHECK=1 python scripts/vllm_infer.py --model_name_or_path zai-org/GLM-4.1V-9B-Thinking --template glm4v --dataset mllm_eval_dataset --adapter_name_or_path /data/shared/jyjang /LLaMA-Factory/saves/glm4_1_v_9b/lora/version020/checkpoint-6000/ --vllm_config '{"max_lora_rank": 32}'

Your output of `DISABLE_VERSION_CHECK=1 python scripts/vllm_infer.py --model_name_or_path zai-org/GLM-4.1V-9B-Thinking --template glm4v --dataset mllm_eval_dataset --adapter_name_or_path /data/shared/jyjang
/LLaMA-Factory/saves/glm4_1_v_9b/lora/version020/checkpoint-6000/ --vllm_config '{"max_lora_rank": 32}'                                                                                                                                                                     ` here

root@dbc3e0868142:/data/shared/jyjang/LLaMA-Factory# CUDA_VISIBLE_DEVICES=0,1 DISABLE_VERSION_CHECK=1 python scripts/vllm_infer.py --model_name_or_path zai-org/GLM-4.1V-9B-Thinking --template glm4v --dataset mllm_eval_dataset --adapter_name_or_path /data/shared/jyjang /LLaMA-Factory/saves/glm4_1_v_9b/lora/version020/checkpoint-6000/ --vllm_config '{"max_lora_rank": 32}' ... ) - video_processor: Glm4vVideoProcessor { "crop_size": null, "data_format": "channels_first", "default_to_square": true, "device": null, "do_center_crop": null, "do_convert_rgb": true, "do_normalize": true, "do_pad": null, "do_rescale": true, "do_resize": true, "do_sample_frames": true, "fps": 2, "image_mean": [ 0.48145466, 0.4578275, 0.40821073 ], "image_std": [ 0.26862954, 0.26130258, 0.27577711 ], "input_data_format": null, "max_image_size": { "longest_edge": 47040000 }, "merge_size": 2, "num_frames": 16, "patch_size": 14, "processor_class": "Glm4vProcessor", "resample": 3, "rescale_factor": 0.00392156862745098, "size": { "longest_edge": 47040000, "shortest_edge": 12544 }, "size_divisor": null, "temporal_patch_size": 2, "video_metadata": null, "video_processor_type": "Glm4vVideoProcessor" }

{
"processor_class": "Glm4vProcessor"
}

                                                                                                                                                                                                                                                                       (

VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] WorkerProc hit an exception. | 1/50 [00:07<06:11, 7.57s/it]
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 824, in _prepare_inputs 17:39:02 [275/1868]
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] self.set_active_loras(self.input_batch, num_scheduled_tokens)
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/v1/worker/lora_model_runner_mixin.py", line 84, in set_active_loras
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] return self._set_active_loras(prompt_lora_mapping, token_lora_mapping,
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/v1/worker/lora_model_runner_mixin.py", line 73, in _set_active_loras
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] self.lora_manager.set_active_adapters(lora_requests, lora_mapping)
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 167, in set_active_adapters
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] WorkerProc hit an exception.
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] set_active_adapters_worker(requests, mapping, self._apply_adapters,
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/adapter_commons/utils.py", line 55, in set_active_adapters_worker
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] Traceback (most recent call last):
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] apply_adapters_func(requests)
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 227, in _apply_adapters
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] self.add_adapter(lora)
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 240, in add_adapter
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] lora = self._load_adapter(lora_request)
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] return func(*args, **kwargs)
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 141, in _load_adapter
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] raise e
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 116, in _load_adapter
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] lora = self._lora_model_cls.from_local_checkpoint(
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] return func(*args, **kwargs)
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/lora/models.py", line 255, in from_local_checkpoint
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] check_unexpected_modules(f)
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1366, in execute_model
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] File "/opt/conda/lib/python3.11/site-packages/vllm/lora/models.py", line 225, in check_unexpected_modules
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] self._prepare_inputs(scheduler_output))
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] raise ValueError(
(VllmWorker rank=1 pid=62315) ERROR 08-01 08:39:02 [multiproc_executor.py:546] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=62314) ERROR 08-01 08:39:02 [multiproc_executor.py:546] ValueError: While loading /data/shared/jyjang/LLaMA-Factory/saves/glm4_1_v_9b/lora/version020/checkpoint-6000/, expected target modules in ['v_proj', 'k_proj', 'up_proj', 'qkv', 'q_proj', '
o_proj', 'down_proj', 'gate_proj', 'proj'] but received ['language_model.model.layers.0.mlp.gate_up_proj', 'language_model.model.layers.0.mlp.gate_up_proj', 'language_model.model.layers.1.mlp.gate_up_proj', 'language_model.model.layers.1.mlp.gate_up_proj', 'language_m
odel.model.layers.10.mlp.gate_up_proj', 'language_model.model.layers.10.mlp.gate_up_proj', 'language_model.model.layers.11.mlp.gate_up_proj', 'language_model.model.layers.11.mlp.gate_up_proj', 'language_model.model.layers.12.mlp.gate_up_proj', 'language_model.model.la
yers.12.mlp.gate_up_proj', 'language_model.model.layers.13.mlp.gate_up_proj', 'language_model.model.layers.13.mlp.gate_up_proj', 'language_model.model.layers.14.mlp.gate_up_proj', 'language_model.model.layers.14.mlp.gate_up_proj', 'language_model.model.layers.15.mlp.g
ate_up_proj', 'language_model.model.layers.15.mlp.gate_up_proj', 'language_model.model.layers.16.mlp.gate_up_proj', 'language_model.model.layers.16.mlp.gate_up_proj', 'language_model.model.layers.17.mlp.gate_up_proj', 'language_model.model.layers.17.mlp.gate_up_proj',
'language_model.model.layers.18.mlp.gate_up_proj', 'language_model.model.layers.18.mlp.gate_up_proj', 'language_model.model.layers.19.mlp.gate_up_proj', 'language_model.model.layers.19.mlp.gate_up_proj', 'language_model.model.layers.2.mlp.gate_up_proj', 'language_mod
el.model.layers.2.mlp.gate_up_proj', 'language_model.model.layers.20.mlp.gate_up_proj', 'language_model.model.layers.20.mlp.gate_up_proj', 'language_model.model.layers.21.mlp.gate_up_proj', 'language_model.model.layers.21.mlp.gate_up_proj', 'language_model.model.layer
s.22.mlp.gate_up_proj', 'language_model.model.layers.22.mlp.gate_up_proj', 'language_model.model.layers.23.mlp.gate_up_proj', 'language_model.model.layers.23.mlp.gate_up_proj', 'language_model.model.layers.24.mlp.gate_up_proj', 'language_model.model.layers.24.mlp.gate
_up_proj', 'language_model.model.layers.25.mlp.gate_up_proj', 'language_model.model.layers.25.mlp.gate_up_proj', 'language_model.model.layers.26.mlp.gate_up_proj', 'language_model.model.layers.26.mlp.gate_up_proj', 'language_model.model.layers.27.mlp.gate_up_proj', 'l
anguage_model.model.layers.27.mlp.gate_up_proj', 'language_model.model.layers.28.mlp.gate_up_proj', 'language_model.model.layers.28.mlp.gate_up_proj', 'language_model.model.layers.29.mlp.gate_up_proj', 'language_model.model.layers.29.mlp.gate_up_proj', 'language_model
.model.layers.3.mlp.gate_up_proj', 'language_model.model.layers.3.mlp.gate_up_proj', 'language_model.model.layers.30.mlp.gate_up_proj', 'language_model.model.layers.30.mlp.gate_up_proj', 'language_model.model.layers.31.mlp.gate_up_proj', 'language_model.model.layers.3
1.mlp.gate_up_proj', 'language_model.model.layers.32.mlp.gate_up_proj', 'language_model.model.layers.32.mlp.gate_up_proj', 'language_model.model.layers.33.mlp.gate_up_proj', 'language_model.model.layers.33.mlp.gate_up_proj', 'language_model.model.layers.34.mlp.gate_up
_proj', 'language_model.model.layers.34.mlp.gate_up_proj', 'language_model.model.layers.35.mlp.gate_up_proj', 'language_model.model.layers.35.mlp.gate_up_proj', 'language_model.model.layers.36.mlp.gate_up_proj', 'language_model.model.layers.36.mlp.gate_up_proj', 'lang
uage_model.model.layers.37.mlp.gate_up_proj', 'language_model.model.layers.37.mlp.gate_up_proj', 'language_model.model.layers.38.mlp.gate_up_proj', 'language_model.model.layers.38.mlp.gate_up_proj', 'language_model.model.layers.39.mlp.gate_up_proj', 'language_model.m$
del.layers.39.mlp.gate_up_proj', 'language_model.model.layers.4.mlp.gate_up_proj', 'language_model.model.layers.4.mlp.gate_up_proj', 'language_model.model.layers.5.mlp.gate_up_proj', 'language_model.model.layers.5.mlp.gate_up_proj', 'language_model.model.layers.6.mlp.
gate_up_proj', 'language_model.model.layers.6.mlp.gate_up_proj', 'language_model.model.layers.7.mlp.gate_up_proj', 'language_model.model.layers.7.mlp.gate_up_proj', 'language_model.model.layers.8.mlp.gate_up_proj', 'language_model.model.layers.8.mlp.gate_up_proj', 'la
nguage_model.model.layers.9.mlp.gate_up_proj', 'language_model.model.layers.9.mlp.gate_up_proj']. Please verify that the loaded LoRA module is correct

🐛 Describe the bug

Title:
ValueError during LoRA inference: Unexpected target modules in checkpoint

Issue Description:
I’m encountering a ValueError during inference using vllm_infer.py when attempting to load a LoRA fine-tuned checkpoint of zai-org/GLM-4.1V-9B-Thinking trained using LLaMA-Factory. The model was trained with the following settings:

lora_target: all
lora_rank: 32
stage: sft
finetuning_type: lora

When launching inference with this command:

CUDA_VISIBLE_DEVICES=0,1 DISABLE_VERSION_CHECK=1 python scripts/vllm_infer.py \
  --model_name_or_path zai-org/GLM-4.1V-9B-Thinking \
  --template glm4v \
  --dataset mllm_eval_dataset \
  --adapter_name_or_path /data/shared/jyjang/LLaMA-Factory/saves/glm4_1_v_9b/lora/version020/checkpoint-6000/ \
  --vllm_config '{"max_lora_rank": 32}'

I receive the following error:

ValueError: While loading /.../checkpoint-6000/, expected target modules in
['v_proj', 'k_proj', 'up_proj', 'qkv', 'q_proj', 'o_proj', 'down_proj', 'gate_proj', 'proj']
but received
['language_model.model.layers.0.mlp.gate_up_proj',
...
'language_model.model.layers.39.mlp.gate_up_proj']

Notes:
• The full list of unexpected modules includes many entries like language_model.model.layers.X.mlp.gate_up_proj (duplicated per layer).
• I suspect this may be due to mismatch in expected LoRA target module names between the vLLM runtime and the way LLaMA-Factory saves adapter weights for this model.
• In LLaMA-Factory, lora_target: all is used, and the model architecture likely names modules differently than the default vLLM expectations.

Environment:
• vLLM version: 0.10.0 / llamafactory: 0.9.4.dev0 / transformers: 4.54.1
• Model: zai-org/GLM-4.1V-9B-Thinking
• Fine-tuning framework: LLaMA-Factory
• Adapter: LoRA checkpoint @ checkpoint-6000
• LoRA Rank: 32
• Launch device: CUDA_VISIBLE_DEVICES=0,1

Request for Help:
• Is there a way to customize or override the expected target module names during LoRA adapter loading in vLLM?
• Alternatively, is there a way to align the naming between LLaMA-Factory and vLLM for LoRA-compatible adapters?
• Would using gate_up_proj as part of lora_target explicitly help here?

Thanks in advance!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

Could you plz provibe your lora config?

[miridih-jyjang]

@jeejeelee , Here is my config!

`model_name_or_path: zai-org/GLM-4.1V-9B-Thinking #/data/shared/checkpoints/hugging_face/GLM-4.1V-9B-Thinking
#model_name_or_path: saves/glm4_1_v_9b/lora/version020/checkpoint-2000
image_max_pixels: 262144
video_max_pixels: 16384
trust_remote_code: true

stage: sft
do_train: true
finetuning_type: lora
lora_rank: 32
lora_target: all
deepspeed: examples/deepspeed/ds_z3_offload_config.json # choices: [ds_z0_config.json, ds_z2_config.json, ds_z3_config.json]

dataset: mllm_dataset #,identity,alpaca_en_demo # video: mllm_video_demo
template: glm4v
cutoff_len: 15000
max_samples: 1000000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

output_dir: saves/glm4_1_v_9b/lora/version060
logging_steps: 10
save_steps: 2000
plot_loss: true
overwrite_output_dir: true
save_only_model: false
save_strategy: steps #
metric_for_best_model: loss #
#save_total_limit: 4
report_to: wandb
run_name: glm4_1_v_9b_lora_sft_v060
load_best_model_at_end: true #

train

per_device_train_batch_size: 2
gradient_accumulation_steps: 2
learning_rate: 1.0e-4
num_train_epochs: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

eval_dataset: mllm_eval_dataset
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 100
do_predict: false
predict_with_generate: false

[miridih-jyjang]

@jeejeelee is there any updates?

[jeejeelee]

Sorry for missing this thread, I mean the adapter_config.json

[miridih-jyjang]

@jeejeelee Oh I see. Here is the adapter_config.json

{
"alpha_pattern": {},
"auto_mapping": null,
"base_model_name_or_path": "zai-org/GLM-4.1V-9B-Thinking",
"bias": "none",
"corda_config": null,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 64,
"lora_bias": false,
"lora_dropout": 0.0,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"qalora_group_size": 16,
"r": 32,
"rank_pattern": {},
"revision": null,
"target_modules": [
"31.mlp.down_proj",
"30.mlp.down_proj",
"36.mlp.down_proj",
"layers.18.mlp.down_proj",
"gate_up_proj",
"layers.19.mlp.down_proj",
"25.mlp.down_proj",
"33.mlp.down_proj",
"layers.23.mlp.down_proj",
"35.mlp.down_proj",
"27.mlp.down_proj",
"layers.12.mlp.down_proj",
"layers.11.mlp.down_proj",
"q_proj",
"26.mlp.down_proj",
"layers.7.mlp.down_proj",
"29.mlp.down_proj",
"layers.6.mlp.down_proj",
"layers.8.mlp.down_proj",
"v_proj",
"layers.1.mlp.down_proj",
"38.mlp.down_proj",
"layers.9.mlp.down_proj",
"24.mlp.down_proj",
"o_proj",
"k_proj",
"layers.20.mlp.down_proj",
"layers.21.mlp.down_proj",
"layers.2.mlp.down_proj",
"layers.10.mlp.down_proj",
"layers.4.mlp.down_proj",
"28.mlp.down_proj",
"layers.3.mlp.down_proj",
"37.mlp.down_proj",
"layers.16.mlp.down_proj",
"32.mlp.down_proj",
"layers.13.mlp.down_proj",
"39.mlp.down_proj",
"layers.0.mlp.down_proj",
"layers.17.mlp.down_proj",
"layers.5.mlp.down_proj",
"layers.22.mlp.down_proj",
"layers.14.mlp.down_proj",
"layers.15.mlp.down_proj",
"34.mlp.down_proj"
],
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}

[jeejeelee]

Thank you for the feedback. I can reproduce this issue, and will try fixing it ASAP

[jeejeelee]

Could you please check #22751 to test if it can fix this issue?

[miridih-jyjang]

@jeejeelee Hi! I checked the #22751 has been merged to vLLM version v0.10.1.1.
For now, Glm4vProcessor is now running, but still got wrong output compared to transformers result.
My library versions are

• transformers: 4.56.1
• vllm: 0.10.1.1

transformers output
Got it, let's analyze the image first. The image shows a cartoon-style illustration of a person fishing. The person is wearing a yellow vest over a white shirt, has dark hair, and is sitting in a blue boat. They're holding a fishing rod that's bent, suggesting they caught something. The background has light blue clouds, giving a calm, outdoor feel.

Now, let's break down each part of the JSON structure.

First, "style": The image is a flat, cartoonish illustration with simple shapes and solid colors. It uses a minimalistic design with clean lines and bright, pastel-like colors. The style is modern vector art, typical of flat design illustrations used in web design or mobile apps.

"core_topic": The core topic is fishing, which is a leisure activity. The image depicts a person engaged in fishing, so the core topic is fishing activity. The genre is likely a flat design illustration, suitable for educational materials, apps, or promotional content related to fishing, outdoor activities, or relaxation.

"core_elements": The main elements are the person (fisherman), the boat, and the fishing rod. These are the primary components that make up the scene. The person is the main subject, the boat provides context (water setting), and the fishing rod is the action element.

Now, let's write the description in Korean, keeping it around 200 tokens, without mentioning white background, and making sure it's descriptive but concise.

For "style": "이미지는 플랫( flat ) 디자인 스타일의 카톡스틱( 카톡스틱 ) 그림으로, 단순한 도형과 실제 색상을 사용합니다. 간결한 라인과 밝은, 과일색 톤의 색상을 통해 현대적인 벡터 아트 스타일을 표현합니다. 단순하고 시각적으로 명확한 디자인으로, 웹 디자인이나 모바일 앱 등에서 자주 사용되는 플랫 디자인 방식입니다."

For "core_topic": "이미지의 핵심 주제는 낚시 활동입니다. 그림에서는 사람이 낚시를 하는 모습을 보여주므로, 낚시와 관련된 휴식적이고 여가적인 활동을 주제로 합니다. 이 이미지는 낚시, 야외 활동, 휴식 등과 같은 주제에 적합하며, 플랫 디자인의 그래픽 아트 장르에 속합니다."

For "core_elements": "이미지를 구성하는 핵심 요소는 낚시하는 사람, 보트, 낚시로드입니다. 사람은 주요 주제를 표현하는 주체이고, 보트는 물 공간의 설정을 제공하며, 낚시로드는 활동적인 요소를 담고 있습니다."

Wait, need to make sure each part is around 200 tokens total? Wait, the total description should be around 200 tokens. Let's check the length.

Now, let's structure the JSON with these descriptions.

First, "style" should analyze the design's aesthetic and color aspects. "core_topic" should explain the main subject and genre. "core_elements" lists the main components.

Let me rephrase each part more concisely.

"style": "이미지는 플랫 디자인 스타일의 카톡스틱 그림으로, 단순한 도형과 실제 색상을 사용합니다. 간결한 라인과 밝은 과일색 톤의 색상을 통해 현대적인 벡터 아트 스타일을 표현합니다. 단순하고 시각적으로 명확한 디자인으로, 웹이나 모바일 앱 등에서 자주 사용되는 플랫 디자인 방식입니다."

"core_topic": "이미지의 핵심 주제는 낚시 활동입니다. 그림에서 사람이 낚시를 하는 모습을 보여주므로, 낚시와 관련된 휴식적이고 여가적인 활동을 주제로 합니다. 이 이미지는 낚시, 야외 활동, 휴식 등과 같은 주제에 적합하며, 플랫 디자인의 그래픽 아트 장르에 속합니다."

"core_elements": "이미지를 구성하는 핵심 요소는 낚시하는 사람, 보트, 낚시로드입니다. 사람은 주요 주제를 표현하는 주체이고, 보트는 물 공간의 설정을 제공하며, 낚시로드는 활동적인 요소를 담고 있습니다."

Now, let's check the total token count. Each part should be around 60-70 tokens, total around 200.

Now, convert this into JSON with proper formatting, no quotes inside strings, no special symbols.

Wait, the JSON structure requires:

• "style" with description
• "core_topic" with description
• "core_elements" with description

All in Korean, no white background, no extra text.

Final JSON:

{
"style": "이미지는 플랫 디자인 스타일의 카톡스틱 그림으로, 단순한 도형과 실제 색상을 사용합니다. 간결한 라인과 밝은 과일색 톤의 색상을 통해 현대적인 벡터 아트 스타일을 표현합니다. 단순하고 시각적으로 명확한 디자인으로, 웹이나 모바일 앱 등에서 자주 사용되는 플랫 디자인 방식입니다.",
"core_topic": "이미지의 핵심 주제는 낚시 활동입니다. 그림에서 사람이 낚시를 하는 모습을 보여주므로, 낚시와 관련된 휴식적이고 여가적인 활동을 주제로 합니다. 이 이미지는 낚시, 야외 활동, 휴식 등과 같은 주제에 적합하며, 플랫 디자인의 그래픽 아트 장르에 속합니다.",
"core_elements": "이미지를 구성하는 핵심 요소는 낚시하는 사람, 보트, 낚시로드입니다. 사람은 주요 주제를 표현하는 주체이고, 보트는 물 공간의 설정을 제공하며, 낚시로드는 활동적인 요소를 담고 있습니다."
}

Wait, let's check if each part is descriptive, accurate, and concise. The style part talks about flat design, cartoonish, simple shapes, colors. Core topic is fishing activity, genre is flat graphic art. Core elements are the main components: person, boat, fishing rod. All without mentioning white background, all in Korean, JSON format with no extra text.{
"style": "이미지는 플랫 디자인 스타일의 카톡스틱 그림으로, 단순한 도형과 실제 색상을 사용합니다. 간결한 라인과 밝은 과일색 톤의 색상을 통해 현대적인 벡터 아트 스타일을 표현합니다. 단순하고 시각적으로 명확한 디자인으로, 웹이나 모바일 앱 등에서 자주 사용되는 플랫 디자인 방식입니다.",
"core_topic": "이미지의 핵심 주제는 낚시 활동입니다. 그림에서 사람이 낚시를 하는 모습을 보여주므로, 낚시와 관련된 휴식적이고 여가적인 활동을 주제로 합니다. 이 이미지는 낚시, 야외 활동, 휴식 등과 같은 주제에 적합하며, 플랫 디자인의 그래픽 아트 장르에 속합니다.",
"core_elements": "이미지를 구성하는 핵심 요소는 낚시하는 사람, 보트, 낚시로드입니다. 사람은 주요 주제를 표현하는 주체이고, 보트는 물 공간의 설정을 제공하며, 낚시로드는 활동적인 요소를 담고 있습니다."
}

vLLM output>

[jeejeelee]

@miridih-jyjang Could you plz provibe your adapter_config.json？

[miridih-jyjang]

@jeejeelee The adapter_config.json that I was using for training lora is as stated above. However, What you asked me might be other adapter_config.json, I don't know what exactly you are asking for. Do you mean arguments for running vllm ?? If so, That is as follows:

cmd = [
    "python", "-m", "vllm.entrypoints.openai.api_server",
    "--model", "zai-org/GLM-4.1V-9B-Thinking",
    "--dtype", "bfloat16",
    "--max-model-len", "2048",
    "--gpu-memory-utilization", "0.90",
    '--limit-mm-per-prompt', '{"image": 1}',
    "--max-num-seqs", "64",
    "--port", "8888",
    "--max-num-batched-tokens", "49152",
    "--enable-prefix-caching",
    "--download-dir", "/data/nas_vol1/huggingface/GLM-4.1V-9B-Thinking"
]

[jeejeelee]

Has LoRA been added to the vision module?