Token indices sequence length

[saboor2632]

I am facing this issue whenever i run my code for scraping bbc or any other site
Error:
Token indices sequence length is longer than the specified maximum sequence length for this model (5102 > 1024). Running this sequence through the model will result in indexing errors

It does not give me complete results

[Qunlexie]

I have this same issue as well especially when working with Ollama LLAMA models

[VinciGit00]

for big websites you should use openai

[PeriniM]

@Qunlexie @saboor2632 there is indeed an issue with the method to calculate chunks for ollama models, in tokenizer_ollama.py. We are using get_num_tokens from langchain lib but it is always limiting the model to 1024 tokens. Even if we change the num of tokens from the graph config using the model_tokens param we could only set it to 1024. It would be better to calculate the num of chunks ourself, using for example openai tokenizer from tiktoken and approximate the num of tokens per chunk. wdyt

[Qunlexie]

We are using get_num_tokens from langchain lib but it is always limiting the model to 1024 tokens. Even if we change the num of tokens from the graph config using the model_tokens param we could only set it to 1024.

Is this a bug that should be raised to langchain? I belive the limit to the tokens is the cause of the issue where only parts of the web page is being retrieved rather all of it.

It would be better to calculate the num of chunks ourself, using for example openai tokenizer from tiktoken and approximate the num of tokens per chunk. wdyt

How would this work? Do you have a practical example? for how this will work with ScrpapeGraph. I really believe that getting Ollama to work properly is key for open source.

Happy to get your thoughts

[Pal-dont-want-to-work]

I face the same problem, hoping author can fix it
Token indices sequence length is longer than the specified maximum sequence length for this model (1548 > 1024). Running this sequence through the model will result in indexing errors