AutocompleteTemplate: Use modelfile template for granite3 #3314
Conversation
granite3 is currently using the "hole filler" template since it lacks FIM support. This template is getting sent to ollama raw, so the template from the model file is not getting applied to it. This commit ensures the proper format is fed to the model, by moving the hole filler system message to be a global constant, and using it in the existing holeFillerTemplate and a new granteHoleFillerTemplate that also applies the appropiate model specific formatting.
✅ Deploy Preview for continuedev ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
this is marked draft because i haven't tested it yet, so haven't confirmed it doesn't regress results. I just sketched it out a few days ago after noticing the problem when looking at ollama debug output. I will test eventually, but for now, I just want to get it off my machine so I don't lose track of it. |
| ) { | ||
| return holeFillerTemplate; | ||
| } | ||
|
|
||
| if (lowerCaseModel.includes("granite3")) { |
There was a problem hiding this comment.
lowerCaseModel.includes("granite3") && !lowerCaseModel.includes("-code")
future proofing the next models that'll support FIM
|
@halfline have you had any time to take a second look at this? |
not yet and the latest granite release has a different template. will get back to this soon |
granite3 is currently using the "hole filler" template since it lacks FIM support. This template is getting sent to ollama raw, so the template from the model file is not getting applied to it.
This commit ensures the proper format is fed to the model, by moving the hole filler system message to be a global constant, and using it in the existing holeFillerTemplate and a new granteHoleFillerTemplate that also applies the appropiate model specific formatting.