Transformers cannot load ModernBERT for sequence classification

[eneko-caf]

System Info

I am trying to test the new ModernBER, following this notebook from the official documentation: https://github.com/AnswerDotAI/ModernBERT/blob/main/examples/finetune_modernbert_on_glue.ipynb model for sequence classification but I am getting the following error:

Traceback (most recent call last):
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1038, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 740, in __getitem__
    raise KeyError(key)
KeyError: 'modernbert'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/train_full_fine_tuning.py", line 75, in <module>
    train_model_ft(tmp_folder_dataset.name, args)
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/train_full_fine_tuning.py", line 39, in train_model_ft
    orchestrate_training_with_epoch_artifacts(dataset=dataset, args=args)
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/utils/training/training_utils.py", line 153, in orchestrate_training_with_epoch_artifacts
    tokenizer, model = get_model_tokenizer(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/utils/training/training_utils.py", line 46, in get_model_tokenizer
    model = AutoModelForSequenceClassification.from_pretrained(training_model_name,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1040, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `modernbert` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

I am using:

• Python version: 3.12.7
• Tranformers version: 4.47.1
• Tranformers information:

- `transformers` version: 4.47.1
- Platform: Linux-6.8.0-1018-aws-x86_64-with-glibc2.35
- Python version: 3.12.7
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:  no
- Using GPU in script?: yes
- GPU type: NVIDIA A10G

The code snippet used is this:

model = AutoModelForSequenceClassification.from_pretrained("answerdotai/ModernBERT-base", cache_dir=model_saving_path,
                                                               num_labels=12, compile=False)

Thank you very much!

Who can help?

@ArthurZucker
@stevhliu

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Just executing

model = AutoModelForSequenceClassification.from_pretrained("answerdotai/ModernBERT-base", cache_dir=model_saving_path,
                                                         num_labels=12, compile=False)

the problem will arise

Expected behavior

To load the model normally

[seanfarr788]

Currently need to compile the latest transformers

pip install git+https://github.com/huggingface/transformers

source: https://huggingface.co/answerdotai/ModernBERT-base/discussions/3

[dzimmerman-nci]

Any idea why I am getting this error during a train() of ModernBert on the latest transformers dev branch?
PyTorch version: 2.4.1+cu118 Transformers version: 4.48.0.dev0

BackendCompilerFailed: backend='inductor' raised: AssertionError: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.clone.default(tensor([...], size=(16,), dtype=torch.uint8), memory_format=torch.contiguous_format)

[dzimmerman-nci]

upgrading to PyTorch version: 2.5.1 seemed to fix the issue

[roei-shlezinger]

I am getting the following error when trying to import transformers.trainer on macOS:

RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):
Failed to import transformers.integrations.integration_utils because of the following error (look up to see its traceback):
Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'CompileConfig' from 'transformers.generation'

I am using Python 3.12.8, PyTorch 2.5.1, and also have bitsandbytes, accelerate, and protobuf installed in my environment.