Skip to content

Conversation

@mudler
Copy link
Owner

@mudler mudler commented Oct 21, 2024

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
  video: "<|video_{{.ID}}|> {{.Text}}"
  image: "<|image_{{.ID}}|> {{.Text}}"
  audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
  multimodal:  "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

  • Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

  • Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

  • Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

@mudler mudler added enhancement New feature or request breaking-change labels Oct 21, 2024
@netlify
Copy link

netlify bot commented Oct 21, 2024

Deploy Preview for localai ready!

Name Link
🔨 Latest commit 6f336da
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

dave-gray101
dave-gray101 previously approved these changes Oct 21, 2024
Copy link
Collaborator

@dave-gray101 dave-gray101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)

@mudler mudler merged commit ccc7cb0 into master Oct 22, 2024
30 of 31 checks passed
@mudler mudler deleted the fix/mm_index branch October 22, 2024 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants