diff --git a/docs/_static/burr_sdlc.png b/docs/_static/burr_sdlc.png new file mode 100644 index 00000000..d27c1155 Binary files /dev/null and b/docs/_static/burr_sdlc.png differ diff --git a/docs/concepts/overview.rst b/docs/concepts/overview.rst index de587899..d9a698e1 100644 --- a/docs/concepts/overview.rst +++ b/docs/concepts/overview.rst @@ -12,3 +12,30 @@ This is a quick overview of Burr's design -- the concepts are explored in more d Note that we did not mention LLMs above at all! That's good -- you want your LLM frameworks to be ever-so-slightly decoupled from them for the best experience (all of AI is just software, plain and simple, after all...). And that's the basics! Let's dive into the details. + +================================ +SDLC with LLM based applications +================================ +If you're building an LLM-based application, you'll want to follow a slightly different SDLC than you would for a traditional software project. Here's a rough outline of what that might look like: + +.. image:: ../_static/burr_sdlc.png + :alt: SDLC with LLMs + :align: center + +The two cycles that exist are: + +1. App Dev Loop. +2. Test Driven Development Loop. + +and you will use one to feed into the other, etc. + +Walking through the diagram the SDLC looks like this: + +1. Write code with Burr. +2. Use Burr's integrated observability, and trace all parts of your application. +3. With the data collected, you can: (1) annotate what was captured and export it, or (2) create a pytest fixture with it. +4. Create a data set from the annotated data or by running tests. +5. Evaluate the data set. +6. Analyze the results. +7. Either adjust code or prompts, or ship the code. +8. Iterate using one of the loops... diff --git a/examples/pytest/README.md b/examples/pytest/README.md index 98d87eab..aaf43401 100644 --- a/examples/pytest/README.md +++ b/examples/pytest/README.md @@ -1,3 +1,17 @@ +# A SDLC with Burr and pytest +Here we show a quick example of a software development lifecycle (SDLC) with Burr and pytest. + +![Burr and pytest](burr_sdlc.png) + +While we don't cover everything in the diagram, in this example we specifically show how to do most of the TDD loop: + +1. Create a test case. +2. Run the test case. +3. Create a dataset. +4. Show how you might construct evaluation logic to evaluate the output of your agent / augmented LLM / application. + +using Burr and pytest. + # Using pytest to evaluate your agent / augmented LLM / application An agent / augmented LLM is a combination of LLM calls and logic. But how do we know if it's working? Well we can test & evaluate it. diff --git a/examples/pytest/burr_sdlc.png b/examples/pytest/burr_sdlc.png new file mode 100644 index 00000000..d27c1155 Binary files /dev/null and b/examples/pytest/burr_sdlc.png differ