Qauntization in glm4-9b failed

[Orion-zhen]

As I run the convert.py with command: CUDA_VISIBLE_DEVICES=1 python convert.py -i /home/orion/ai/Models/glm4-9b -o ./tmp-file -cf /home/orion/ai/Models/glm4-9b-4-exl2 -r 256, it runs into an error saying TypeError: Value for eos_token_id is not of expected type <class 'int'>.

It seems that the architecture of glm4 hasn't been supported yet.

Steps to reproduce: Just download the glm4-9b model and run the convert.py as README says.

Full console log:

 !! Warning, unknown architecture: ChatGLMModel
 !! Loading as LlamaForCausalLM
Traceback (most recent call last):
  File "/home/orion/repo/exllamav2/convert.py", line 71, in <module>
    config.prepare()
  File "/home/orion/repo/exllamav2/exllamav2/config.py", line 187, in prepare
    self.eos_token_id = read(read_config, int, "eos_token_id", None)  # 2
  File "/home/orion/repo/exllamav2/exllamav2/config.py", line 40, in read
    raise TypeError(f"Value for {key} is not of expected type {expected_type}")
TypeError: Value for eos_token_id is not of expected type <class 'int'>

[turboderp]

Yeah, the architecture isn't supported. There's a bunch of little things that would have to be updated, like how the EOS token is a list all of a sudden, scaled attention layers and such. It's not high on the list of priorities at the moment. Not sure if the model is any good, or if it's any good without the multimodal capabilities which wouldn't be supported anyway.

[Orion-zhen]

As the THUDM declares, the glm4-9b could out-perform llama3-8b. So it might worth a try. BTW, would it be possible to add multimodal support to exllamav2 in the future? I see that multimodal llm (vlm) could be the next trend.

[turboderp]

Multimodal is possible, of course, as is GLM4 in general, along with diffusion models, TTS, you name it. I just have to prioritize. But contributions are always welcome.

[Trapper4888]

It's getting old so there will probably be a better glm model soon, maybe esier to integrate into exl2, but it is supposed to be SOTA in a hallucination(less) benchmark:
https://huggingface.co/spaces/vectara/leaderboard

It would be cool for the dev of exotic architectures to contribute to their integration into inference engine, but I'm not blaming them either First I'm not contributing much myself, plus things move so fast, and its good that people do what they are best at. If there would be a very high pressure to have glm exl2, someone would eventually do it.

edit: Seems like it's getting hfized: huggingface/transformers#33823

[randoentity]

exllamav2/attn.py(1469)forward_torch() 1468 import('ipdb').set_trace() -> 1469 raise Exception from e 1470 ext_c.rope_(key_states, sin, cos, past_len_rope, num_key_value_heads, head_dim, position_offsets, self.archparams.rope_style == RopeStyle.NEOX)

ipdb> cos.size()
torch.Size([1, 1, 32768, 64])
ipdb> head_dim
128

ipdb> c

Traceback (most recent call last): File "/home/kevin/repos/exllamav2/exllamav2/attn.py", line 1466, in forward_torch

RuntimeError: cos table does not match head_dim

Changing 128 to 64 in config.json doesn't work.

Changing head_dim inline allows conversion to continue, probably results in a broken quant, but I'll try it out.

Update: gibberish as expected.

Update: fresh env, conversion running without the error.

Update: conversion succeeded, but I'm getting gibberish.

olanolanREA十字olan.react Hess тушolan.react或缺.reactelas тушelas
тушolan或缺十字olanREA十字十字reaREAREA.reactrea.reactolan.react或缺.react
туш十字.react.react.react.react.dest туш тушREA或缺十字
тушrearea十字reaolan十字elasrea.dest.reactrea

Using chatml.jinja because:

ERROR: getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False) ERROR: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: jinja2.exceptions.UndefinedError: 'endpoints.OAI.types.chat_completion.ChatCompletionMessage object' has no attribute 'get' ERROR: Sent to request: TemplateError: 'endpoints.OAI.types.chat_completion.ChatCompletionMessage object' has no attribute 'get'