Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(templates): use a single template for multimodals messages #3892

Merged
merged 1 commit into from
Oct 22, 2024

Conversation

mudler
Copy link
Owner

@mudler mudler commented Oct 21, 2024

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
  video: "<|video_{{.ID}}|> {{.Text}}"
  image: "<|image_{{.ID}}|> {{.Text}}"
  audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
  multimodal:  "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

  • Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

  • Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

  • Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

@mudler mudler added enhancement New feature or request breaking-change labels Oct 21, 2024
Copy link

netlify bot commented Oct 21, 2024

Deploy Preview for localai ready!

Name Link
🔨 Latest commit 6f336da
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

dave-gray101
dave-gray101 previously approved these changes Oct 21, 2024
Copy link
Collaborator

@dave-gray101 dave-gray101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)

@mudler mudler merged commit ccc7cb0 into master Oct 22, 2024
30 of 31 checks passed
@mudler mudler deleted the fix/mm_index branch October 22, 2024 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants