You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While developing LLamaWorker, I used InteractiveExecutor and ChatHistory. During testing with qwen2, I noticed that the responses often ended with a strange character Ċ.
Later, to facilitate the addition of function_call and to avoid the strict constraints of ChatHistory, I tried using InteractiveExecutor directly. However, during testing, I observed that the stop words handling seemed to have issues:
It includes an unexpected stop marker <|im_start|> in addition to the strange character Ċ.
Subsequently, I tried using StatelessExecutor, which seemed to provide perfect results. However, for each inference request, the following log entries were printed twice:
This seems unnecessary. Additionally, after using StatelessExecutor, its underlying Context cannot be used for token counting because it has been disposed of. Would it be beneficial for ILLamaExecutor to have a property like PromptTokens to reflect the number of input tokens?
Description
While developing LLamaWorker, I used
InteractiveExecutor
andChatHistory
. During testing with qwen2, I noticed that the responses often ended with a strange characterĊ
.You can view the related code here:
LLamaWorker v1.0.38 - LLmModelService.cs
Later, to facilitate the addition of
function_call
and to avoid the strict constraints ofChatHistory
, I tried usingInteractiveExecutor
directly. However, during testing, I observed that the stop words handling seemed to have issues:It includes an unexpected stop marker
<|im_start|>
in addition to the strange characterĊ
.Subsequently, I tried using
StatelessExecutor
, which seemed to provide perfect results. However, for each inference request, the following log entries were printed twice:Upon examining the source code, I found that
StatelessExecutor
creates and immediately disposes of aContext
:This seems unnecessary. Additionally, after using
StatelessExecutor
, its underlyingContext
cannot be used for token counting because it has been disposed of. Would it be beneficial forILLamaExecutor
to have a property likePromptTokens
to reflect the number of input tokens?You can view the related code here:
LLamaWorker - MyStatelessExecutor.cs
Therefore
StatelessExecutor
appears to have some redundant code, potentially reducing processing efficiency.The text was updated successfully, but these errors were encountered: