[New Model]: Chameleon support

[nopperl]

The model to consider.

https://huggingface.co/facebook/chameleon
(as of now, the models can be downloaded using the model form)

Chameleon is an interesting multimodal model architecture based on Llama 2. It adds image inputs and outputs to Llama 2 by tokenizing images using a VQ-VAE and adding the codebook to Llama's tokenizer vocabulary.
In principle, it supports text and images as input and output in arbitrary combination. However, the released models were finetuned to prevent image generation.

The closest model vllm already supports.

LlamaForCausalLM

What's your difficulty of supporting the model you want?

For text->text support, the implementation should actually be fairly easy. The model is based on Llama-2 with the following differences:

• QK norm
• reordering the norm similar to Swin Transformer (normalizing the outputs of the attention and ffn blocks instead of the inputs)

To enable image inputs, image tokenization using the provided VQ-VAE needs to be added.

Further info:

• Original paper: https://arxiv.org/abs/2405.09818
• Reference implementation: https://github.com/facebookresearch/chameleon

[mgoin]

Apparently there is ongoing work by HF to get this architecture into transformers huggingface/transformers#31505
Once that is implementation is made we can look at porting it over.

[ywang96]

Chameleon PR ontransformers to track huggingface/transformers#31534

[ywang96]

I made a PR #5770 based off the transformers PR as an initial text-only support for this model and will wait for HuggingFace to release the weights to verify the implementation.

For the VQVAE - I plan to add it in next PR if that makes sense.

[mgoin]

Looks like they landed it today!

[ywang96]

@nopperl The two chameleon PRs have been merged and will be included in the upcoming 0.5.3 release, so please give it a try and let me know if you run into any issue!

For now it only supports one image per request, but multi-image is definitely on our Q3 roadmap!

I spoke too early - it turned out we still need transformers to make a release since we do use the image processor for Chameleon but it's not included in vLLM (and I don't think we should include it anyways)

Huggingface has made a new release so this model should work on 0.5.3.post1 docker image!