Error: Qwen 3 Coder inference bug

[ronaldmannak]

Even though Qwen 3 and Qwen 3 MOE work without issue, Qwen 3 Coder doesn't seem to work, see issue and logs here:

See: PicoMLX/PicoMLXServer#7 (comment)

[davidkoski]

These are the errors:

2025-08-04T02:33:54Z error: Error generating: Runtime error: Cannot get length of type: UndefinedValue
2025-08-04T02:33:54Z error: Error in generation: Runtime error: Cannot get length of type: UndefinedValue

and it looks like that is coming from the template (Jinja):

throw JinjaError.runtime("Cannot get length of type: \(type(of: arg))")

I don't know if that is an error in the template or the swift-jinja implementation or the caller (mlx-swift-examples).

[ronaldmannak]

Good catch. I'm unable to look into more detail myself right now but did ping John

[johnmai-dev]

Good catch. I'm unable to look into more detail myself right now but did ping John

Thank you for the feedback. I will check it out after work.

[johnmai-dev]

The error comes from the template.

{%- if message.role == "assistant" and message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}

[johnmai-dev]

My MacBook Pro doesn't have enough memory to run Qwen3 Coder. Could you help me verify this PR? Thank you very much.@ronaldmannak
huggingface/swift-jinja#20

[ronaldmannak]

@johnmai-dev I'm on a crappy hotel network. The good news: the Jinja error seems to be gone. The bad news: HubAPI is throwing some errors, probably because it doesn't like crappy hotel networks. I'll be able to fully test it in about 10 hours. Sorry for the delay.

2025-08-04T16:36:58Z trace: Loading LLM model: mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit
2025-08-04T16:37:14Z error: Error loading mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit: Offline mode error: Metadata not available for download_date.txt
2025-08-04T16:37:14Z error: Unable to generate completion: Offline mode error: Metadata not available for download_date.txt

[ronaldmannak]

Thanks @johnmai-dev I tested both 4bit and 4bit-DWQ and both work. The error I encountered earlier is Swift-Transformers issue (It looks like the latest Transformers update completely broke my app but that has nothing to do with the Qwen 3 Coder issue). Closing the issue.

[johnmai-dev]

Release Jinja 1.2.2 https://github.com/johnmai-dev/Jinja/releases/tag/1.2.2