installing the dependencies for layoutlmv3, the tokenizers library throws an error

[Jorya2333]

Describe the bug
Model I am using ( LayoutLMv3):

The problem arises when using:

• [1] the official example scripts: (give details below)
• my own modified scripts: (give details below)

A clear and concise description of what the bug is.

When installing the dependencies for layoutlmv3, the tokenizers library throws an error. I found some information suggesting that the issue is related to an improperly configured Rust compilation environment, but I already have Rust version rustc 1.82.0 installed on my machine. The error is as follows:

error: cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- failed with code 101
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tokenizers
Successfully built seqeval deepspeed
Failed to build tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

To Reproduce
Steps to reproduce the behavior:

1. Clone the layoutlmv3 repository
2. Run pip install -r requirements.txt
3. Encounter a Rust compilation error, preventing the installation of the tokenizers library

Expected behavior
A clear and concise description of what you expected to happen.

• Platform:Ubuntu 22.04
• Python version: 3.12, 3.11, 3.10 (all versions have issues)
• PyTorch version (GPU?): 12.4