firefunction-v2 infinite loop #56
Comments
|
|
Notice this example. In the console I actually DO see it list the contents of the directory. Nevertheless, in the web UI, the agent claims: "The current directory is empty. Please provide more information about the task you want to accomplish." 
Console log: |
|
BTW, the reason I'm interested in Firefunction right now is because it's on the Fireworks API for FREE. |
|
This one actually worked because I stopped tripping it up with the comments about what tools to use: 
But I still thought you'd want to see the log: ^^^ What's going on up here with the "RESPOND" call? |
|
The function call syntax seems fine. The reason for the repeated calls is likely due to one scenario: the file_researcher keeps returning some invalid nonsense, causing the main agent to retry repeatedly. Many open-source models have a strong tendency to repeat previous dialogue history, even if it has been proven to be incorrect. In the second issue, although the LLM saw the result of the ls command, it still thought the directory was empty, which is caused by hallucination, indicating poor reasoning ability of the LLM. Usually, it is difficult to improve the performance of such LLMs through prompt engineering. The final issue with the RESPOND function arose because the main agent saw the history of other agents using the RESPOND function from long-term memory and imitated the call, which is explicitly forbidden in the system prompt. This also stems from the LLM's inadequate reasoning ability. Overall, I personally feel that this LLM's capabilities are insufficient to support AIlice's operation, especially since listing files in a directory is the simplest task. Another reason might be that AIlice's function call mechanism is entirely different from other agents' JSON-based call mechanisms. Thus, although the model was fine-tuned for JSON syntax function calls, these fine-tunings did not genuinely improve AIlice's execution. |
|
I do like your angle of just leap-frogging the function call format with your own format that can just work with every model. |
|
Although AIlice widely uses a "standard" function call syntax, it actually supports more user-customized function call formats (but due to the limitations of regular expressions, we may not be able to achieve full JSON call functionality). For these agents that perform special subtasks, we can create a new type of agent specifically for them, define a new function call syntax within it, and configure it to the corresponding LLM. This is indeed achievable. However, I think it's important to explain here the reasons for initially choosing not to make targeted adjustments for specific models. LLM development is progressing too rapidly. Any software designed for today's models is likely to face obsolescence by the time it's released. Therefore, AIlice is designed for the future models. We assume that future models will certainly resolve issues like syntax errors in function calls present in current models and bring more powerful reasoning capabilities. As such, AIlice's prompt design focuses more on what a clear and complete expression should be, rather than on what kind of prompt would yield better results for a specific model like GPT-4. This strategy greatly reduces the burden on developers. We only write the most rational prompts and then wait for LLM development to catch up, rather than getting bogged down in endless prompt engineering. Facts have proven this strategy effective. For leading models, correctly executing these prompts is effortless, and with the development of open-source models, many open-source LLMs have emerged that can understand and execute these prompts well. |
I'm only posting this because the contents of the infinite loop make me think it should have worked. It seems like the function call is formed properly, so I don't know why it just repeats over and over again unless it's not seeing the output or something?
The text was updated successfully, but these errors were encountered: