Merge pull request #873 from ScrapeGraphAI/refactoring-examples

Refactoring examples
ScrapeGraphAI · Jan 12, 2025 · 9ae975e · 9ae975e
2 parents 1d17d92 + c4dba42
commit 9ae975e
Show file tree

Hide file tree

Showing 527 changed files with 1,650 additions and 24,041 deletions.
diff --git a/cookbook/README.md b/cookbook/README.md
diff --git a/docs/source/getting_started/installation.rst b/docs/source/getting_started/installation.rst
@@ -25,18 +25,18 @@ The library is available on PyPI, so it can be installed using the following com
 
    It is higly recommended to install the library in a virtual environment (conda, venv, etc.)
 
-If your clone the repository, it is recommended to use a package manager like `rye <https://rye.astral.sh/>`_.
-To install the library using rye, you can run the following command:
+If your clone the repository, it is recommended to use a package manager like `uv <https://github.com/astral-sh/uv>`_.
+To install the library using uv, you can run the following command:
 
 .. code-block:: bash
 
-   rye pin 3.10
-   rye sync
-   rye build
+   uv pin 3.10
+   uv sync
+   uv build
 
 .. caution::
 
-      **Rye** must be installed first by following the instructions on the `official website <https://rye.astral.sh/>`_.
+      **Rye** must be installed first by following the instructions on the `official website <https://github.com/astral-sh/uv>`_.
 
 Additionally on Windows when using WSL
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

diff --git a/docs/source/introduction/overview.rst b/docs/source/introduction/overview.rst
@@ -30,37 +30,93 @@ ScrapGraphAI supports a wide range of AI models from various providers. Each mod
 OpenAI Models
 -------------
 - GPT-3.5 Turbo (16,385 tokens)
-- GPT-4 (8,192 tokens)
+- GPT-3.5 (4,096 tokens)
+- GPT-3.5 Turbo Instruct (4,096 tokens)
 - GPT-4 Turbo Preview (128,000 tokens)
-- GPT-4o (128000 tokens)
-- GTP-4o-mini (128000 tokens)
+- GPT-4 Vision Preview (128,000 tokens)
+- GPT-4 (8,192 tokens)
+- GPT-4 32k (32,768 tokens)
+- GPT-4o (128,000 tokens)
+- O1 Preview (128,000 tokens)
+- O1 Mini (128,000 tokens)
 
 Azure OpenAI Models
 -------------------
 - GPT-3.5 Turbo (16,385 tokens)
-- GPT-4 (8,192 tokens)
+- GPT-3.5 (4,096 tokens)
 - GPT-4 Turbo Preview (128,000 tokens)
-- GPT-4o (128000 tokens)
-- GTP-4o-mini (128000 tokens)
+- GPT-4 (8,192 tokens)
+- GPT-4 32k (32,768 tokens)
+- GPT-4o (128,000 tokens)
+- O1 Preview (128,000 tokens)
+- O1 Mini (128,000 tokens)
 
 Google AI Models
 ----------------
 - Gemini Pro (128,000 tokens)
+- Gemini 1.5 Flash (128,000 tokens)
 - Gemini 1.5 Pro (128,000 tokens)
+- Gemini 1.0 Pro (128,000 tokens)
 
 Anthropic Models
 ----------------
 - Claude Instant (100,000 tokens)
-- Claude 2 (200,000 tokens)
+- Claude 2 (9,000 tokens)
+- Claude 2.1 (200,000 tokens)
 - Claude 3 (200,000 tokens)
+- Claude 3.5 (200,000 tokens)
+- Claude 3 Opus (200,000 tokens)
+- Claude 3 Sonnet (200,000 tokens)
+- Claude 3 Haiku (200,000 tokens)
 
 Mistral AI Models
 -----------------
-- Mistral Large (128,000 tokens)
+- Mistral Large Latest (128,000 tokens)
+- Open Mistral Nemo (128,000 tokens)
+- Codestral Latest (32,000 tokens)
 - Open Mistral 7B (32,000 tokens)
 - Open Mixtral 8x7B (32,000 tokens)
+- Open Mixtral 8x22B (64,000 tokens)
+- Open Codestral Mamba (256,000 tokens)
 
-For a complete list of supported models and their token limits, please refer to the API documentation.
+Ollama Models
+-------------
+- Command-R (12,800 tokens)
+- CodeLlama (16,000 tokens)
+- DBRX (32,768 tokens)
+- DeepSeek Coder 33B (16,000 tokens)
+- Llama2 Series (4,096 tokens)
+- Llama3 Series (8,192-128,000 tokens)
+- Mistral Models (32,000-128,000 tokens)
+- Mixtral 8x22B Instruct (65,536 tokens)
+- Phi3 Series (12,800-128,000 tokens)
+- Qwen Series (32,000 tokens)
+
+Hugging Face Models
+------------------
+- Grok-1 (8,192 tokens)
+- Meta Llama 3 Series (8,192 tokens)
+- Google Gemma Series (8,192 tokens)
+- Microsoft Phi Series (2,048-131,072 tokens)
+- GPT-2 Series (1,024 tokens)
+- DeepSeek V2 Series (131,072 tokens)
+
+Bedrock Models
+-------------
+- Claude 3 Series (200,000 tokens)
+- Llama2 & Llama3 Series (4,096-8,192 tokens)
+- Mistral Series (32,768 tokens)
+- Titan Embed Text (8,000 tokens)
+- Cohere Embed (512 tokens)
+
+Fireworks Models
+---------------
+- Llama V2 7B (4,096 tokens)
+- Mixtral 8x7B Instruct (4,096 tokens)
+- Llama 3.1 Series (131,072 tokens)
+- Mixtral MoE Series (65,536 tokens)
+
+For a complete and up-to-date list of supported models and their token limits, please refer to the API documentation.
 
 Understanding token limits is crucial for optimizing your scraping tasks. Larger token limits allow for processing more text in a single API call, which can be beneficial for scraping lengthy web pages or documents.
 
@@ -139,3 +195,8 @@ Sponsors
    :width: 15%
    :alt: Stat Proxies
    :target: https://dashboard.statproxies.com/?refferal=scrapegraph
+
+.. image:: ../../assets/scrapedo.png
+   :width: 11%
+   :alt: Scrapedo
+   :target: https://scrape.do
diff --git a/docs/source/modules/scrapegraphai.helpers.models_tokens.rst b/docs/source/modules/scrapegraphai.helpers.models_tokens.rst
@@ -19,7 +19,7 @@ Example usage:
    print(f"GPT-4 token limit: {gpt4_limit}")
 
    # Check the token limit for a specific model
-   model_name = "gpt-3.5-turbo"
+   model_name = "gpt-4o-mini"
    if model_name in models_tokens['openai']:
        print(f"{model_name} token limit: {models_tokens['openai'][model_name]}")
    else:

diff --git a/docs/source/scrapers/benchmarks.rst b/docs/source/scrapers/benchmarks.rst
diff --git a/examples/ScrapegraphAI_cookbook.ipynb b/examples/ScrapegraphAI_cookbook.ipynb
diff --git a/examples/anthropic/.env.example b/examples/anthropic/.env.example
diff --git a/examples/anthropic/code_generator_graph_anthropic.py b/examples/anthropic/code_generator_graph_anthropic.py
diff --git a/examples/anthropic/csv_scraper_anthropic.py b/examples/anthropic/csv_scraper_anthropic.py
diff --git a/examples/anthropic/csv_scraper_graph_multi_anthropic.py b/examples/anthropic/csv_scraper_graph_multi_anthropic.py