Update integrations for 0.9.1

bosun-ai · Sep 1, 2024 · 6e233be · 6e233be
1 parent 51c4b4e
commit 6e233be
Show file tree

Hide file tree

Showing 4 changed files with 23 additions and 17 deletions.
diff --git a/src/content/docs/concepts/loading-data.md b/src/content/docs/concepts/loading-data.md
@@ -27,9 +27,10 @@ Indexing pipelines can also be created from an indexing stream with `Pipeline::f
 
 <small>
 
-| Name           | Description                                                         | Feature Flag |
-| -------------- | ------------------------------------------------------------------- | ------------ |
-| FileLoader     | Loads files with an optional extension filter, respecting gitignore |              |
-| ScrapingLoader | Scrapes a website using the `spider` crate                          | scraping     |
+| Name           | Description                                                                     | Feature Flag |
+| -------------- | ------------------------------------------------------------------------------- | ------------ |
+| FileLoader     | Loads files with an optional extension filter, respecting gitignore             |              |
+| ScrapingLoader | Scrapes a website using the `spider` crate and html to markdown transformations | scraping     |
+| fluvio         | Load data directly from fluvio streams                                          | fluvio       |
 
 </small>
diff --git a/src/content/docs/concepts/prompting-embedding.md b/src/content/docs/concepts/prompting-embedding.md
@@ -35,12 +35,13 @@ Or in human language: "Given a list of things to Embed, give me embeddings". The
 
 <small>
 
-| Name                       | Description                                               | Feature Flag |
-| -------------------------- | --------------------------------------------------------- | ------------ |
-| OpenAI                     | Implements both SimplePrompt and Embed via `async_openai` | openai       |
-| FastEmbed                  | Implements Embed via `fastembed-rs`                       | fastembed    |
-| Titan (Bedrock)            | AWS Titan via Bedrock                                     | aws-bedrock  |
-| Anthropic models (Bedrock) | Recen Anthropic models via Bedrock                        | aws-bedrock  |
-| Groq                       | Implements `SimplePrompt`                                 | groq         |
+| Name                       | Description                                                                 | Feature Flag |
+| -------------------------- | --------------------------------------------------------------------------- | ------------ |
+| OpenAI                     | Implements both SimplePrompt and Embed via `async_openai`                   | openai       |
+| FastEmbed                  | Implements Embed via `fastembed-rs`, including many dense and sparse models | fastembed    |
+| Titan (Bedrock)            | AWS Titan via Bedrock                                                       | aws-bedrock  |
+| Anthropic models (Bedrock) | Recen Anthropic models via Bedrock                                          | aws-bedrock  |
+| Groq                       | Implements `SimplePrompt`                                                   | groq         |
+| ollama                     | Local models via ollama.rs implementing SimplePrompt                        | ollama       |
 
 </small>
diff --git a/src/content/docs/concepts/storing-results.md b/src/content/docs/concepts/storing-results.md
@@ -30,10 +30,11 @@ If `batch_size` is implemented for the storage, the stream will always prefer `b
 
 <small>
 
-| Name          | Description                                          | Feature Flag |
-| ------------- | ---------------------------------------------------- | ------------ |
-| Redis         | Persists nodes by default as json                    | redis        |
-| Qdrant        | Persists nodes in qdrant; expects a vector to be set | qdrant       |
-| MemoryStorage | Persists nodes in memory; great for debugging        |              |
+| Name          | Description                                   | Feature Flag |
+| ------------- | --------------------------------------------- | ------------ |
+| Redis         | Persists nodes by default as json             | redis        |
+| Qdrant        | Persists nodes in qdrant                      | qdrant       |
+| MemoryStorage | Persists nodes in memory; great for debugging |              |
+| LanceDB       | Persist and retrieve in lancedb               | lancedb      |
 
 </small>
diff --git a/src/content/docs/getting-started/features.mdx b/src/content/docs/getting-started/features.mdx
@@ -11,10 +11,11 @@ import { Icon } from "@astrojs/starlight/components";
 
 - Fast streaming indexing pipeline with async, parallel processing
 - Experimental query pipeline
-- Integrations with OpenAI, Groq, Redis, Qdrant, FastEmbed, and Treesitter
+- Integrations with OpenAI, Groq, Redis, Qdrant, FastEmbed, Treesitter, and more
 - A variety of loaders, transformers, embedders, and other common, generic tools
 - Bring your own transformers by extending straightforward traits
 - Jinja-like templating for prompts
+- Evaluate pipelines with RAGAS
 - Splitting and merging pipelines
 - Store into multiple backends
 
@@ -47,6 +48,8 @@ For all our integrations and tools we aim to provide sane defaults to get you st
 | Redis         | redis        | Supports caching and storage                                      |
 | Spider & htmd | scraping     | Scrape websites fast and convert the html to markdown             |
 | Treesitter    | tree-sitter  | Code splitting and various transformers to effectively index code |
+| Fluvio        | fluvio       | Loading data from fluvio streams                                  |
+| Lancedb       | lancedb      | Storing and retrieval from lancedb                                |
 
 :::note
 Swiftide is under active development and might have more features than specified here, for an up to date list check out our api documentation [docs.rs](https://docs.rs/crate/swiftide/latest/)