Skip to content

Conversation

@mudler
Copy link
Owner

@mudler mudler commented Oct 4, 2024

Description

This PR allows to customize the placeholders for the images that are sent back to the model.

For context, some models/libraries have different way to express images, videos or audio placeholders. For example, llama.cpp backend expects images within an [img-ID] tag, but other backends/models (e.g. vLLM) use a different notation ( <|image_|>). This changeset allow to specify the notation in the template config section of the YAML file:

For example, to override defaults, now it is possible to set in the model configuration the following:

template:
  video: "<|video_{{.ID}}|> {{.Text}}"
  image: "<|image_{{.ID}}|> {{.Text}}"
  audio: "<|audio_{{.ID}}|> {{.Text}}"

Notes for Reviewers

This PR is related to #3670 #3602 #2318

Signed commits

  • Yes, I signed my commits.

@mudler mudler added the enhancement New feature or request label Oct 4, 2024
@netlify
Copy link

netlify bot commented Oct 4, 2024

Deploy Preview for localai ready!

Name Link
🔨 Latest commit 03b3ce9
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/670006cb5994f80008c5e9ad
😎 Deploy Preview https://deploy-preview-3728--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mudler mudler changed the title feat(multimodal): allow to template image placeholders feat(multimodal): allow to template placeholders Oct 4, 2024
@mudler mudler merged commit 648ffdf into master Oct 4, 2024
32 of 33 checks passed
@mudler mudler deleted the feat/multimodal_template branch October 4, 2024 16:32
siddimore pushed a commit to siddimore/LocalAI that referenced this pull request Oct 6, 2024
feat(multimodal): allow to template image placeholders

Signed-off-by: Ettore Di Giacinto <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants