JSON as lingua franca

[andraz]

tl;dr: I propose we adopt JSON as the "lingua franca" for all I/O traffic as core strategy.

Primarily for LLM responses and also any "bus" traffic we log/buffer for later human or LLM consumption.

JSON can also become "food for vectors" because of its structured nature, which provides a long-term advantage in clustering.

Combining the strengths of the following approaches at the lowest possible architectural level will, in my opinion, create a highly resilient system for building our apps.

JSON

JSON is a standardized, lightweight, and human-readable format that is widely supported by programming languages and frameworks. This makes it an ideal choice for exchanging data between different systems, including LLMs and our applications.

Some of the key benefits of using JSON for LLM responses include:

• Ease of use: JSON is easy to parse and interpret, both for humans and machines. This makes it easy to integrate LLM responses into our applications and to share them between different systems.
• Efficiency: JSON is an efficient format for transmitting and storing data. This is important for applications that need to process large volumes of LLM responses or that need to exchange data with LLMs over a network.
• Flexibility: JSON is a flexible format that can be used to represent a wide variety of data structures. This makes it well-suited for representing the complex and diverse responses that can be generated by LLMs.
• Wide support: JSON is widely supported by programming languages and frameworks. This makes it easy to find developers who can work with JSON data, and it makes it easy to integrate LLM responses into a wide variety of applications.

Langchain solution:

https://python.langchain.com/docs/modules/model_io/output_parsers/pydantic

The Pydantic parser can help you to generate JSON data that is schema-valid and consistent. It hels to improve the quality of LLM outputs by providing them with a schema to follow. This can help to reduce the number of errors and inconsistencies in LLM outputs.

Other output parsers (like Retry parser) can also help by "healing" the response by passing the schema and broken response back to LLM to generate the same response with correct schema.

Grammar solution:

Llama-cpp is a powerful LLM that can be used to generate text, translate languages, and answer questions. However, sometimes you need to control the output of your LLM to ensure that it matches a specific format, style, or grammar. This is where Llama-cpp Python grammars come in.

Llama-cpp Python grammars allow you to specify a grammar that your LLM must follow when generating text. This can be useful for a variety of tasks, such as:

• Generating code in a specific programming language
• Generating data that conforms to a specific schema (like JSON)

https://archive.is/Tohz8

Algorithmic solution:

JSONRepair is a tool that can be used to repair invalid JSON objects, providing multiple benefits:

• Improves the reliability of your JSON parsing by repairing invalid JSON objects. This is important for applications that need to be able to parse JSON data from a variety of sources, including LLMs.
• Reduces the number of errors that occur when parsing JSON data. This is important for applications that need to be able to handle invalid JSON data gracefully.
• Improves the performance of your JSON parsing by repairing invalid JSON objects before they are parsed. This is important for applications that need to parse large volumes of JSON data.
• Enhances the robustness of your JSON parsing applications by making them more tolerant of invalid JSON data.

https://github.com/josdejong/jsonrepair

Splitting issue

JSON can get corrupted if LLM runs out of tokens while it is responding. In the chat interface we are able to prompt the LLM to continue with its response in the next request.

In terms of the architecture itself, how should this edge case be handled?

[daveshap]

YAML is more optimized for being human readable, but I don't think we're going to require a single structured markup. We need to be markup agnostic. English is the lingua franca on the buses as far as I'm concerned. That being said, this is a good time for experimentation.

[thomafred]

If a may be so bold - a combination of langchain, pydantic and the agent protocol could be a very good combination for the layers. Objects on the busses can for example be expressed as agentprotocol tasks.