-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
ec9ef2b
commit 11ae717
Showing
3 changed files
with
71 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,3 +5,6 @@ scrapegraphai | |
:maxdepth: 4 | ||
|
||
scrapegraphai | ||
|
||
scrapegraphai.helpers.models_tokens | ||
|
28 changes: 28 additions & 0 deletions
28
docs/source/modules/scrapegraphai.helpers.models_tokens.rst
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
scrapegraphai.helpers.models_tokens module | ||
========================================== | ||
|
||
.. automodule:: scrapegraphai.helpers.models_tokens | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
This module contains a comprehensive dictionary of AI models and their corresponding token limits. The `models_tokens` dictionary is organized by provider (e.g., OpenAI, Azure OpenAI, Google AI, etc.) and includes various models with their maximum token counts. | ||
|
||
Example usage: | ||
|
||
.. code-block:: python | ||
from scrapegraphai.helpers.models_tokens import models_tokens | ||
# Get the token limit for GPT-4 | ||
gpt4_limit = models_tokens['openai']['gpt-4'] | ||
print(f"GPT-4 token limit: {gpt4_limit}") | ||
# Check the token limit for a specific model | ||
model_name = "gpt-3.5-turbo" | ||
if model_name in models_tokens['openai']: | ||
print(f"{model_name} token limit: {models_tokens['openai'][model_name]}") | ||
else: | ||
print(f"{model_name} not found in the models list") | ||
This information is crucial for users to understand the capabilities and limitations of different AI models when designing their scraping pipelines. |