How to use ViTPose model in mmpose?

[seon-creator]

I saw the previous question for using vitpose, but I didn't understand how to follow the installation guide.
Which directory should I install the mmpretrain?
If installed successfully, how to register the VisionTransformer?

The Vitpose config file root is
mmpose/configs/body_2d_keypoint/topdown_heatmap/coco/td-hm_ViTPose-base_8xb64-210e_coco-256x192.py

The model setting like this

# model settings
model = dict(
    type='TopdownPoseEstimator',
    data_preprocessor=dict(
        type='PoseDataPreprocessor',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        bgr_to_rgb=True),
    backbone=dict(
        type='mmpretrain.VisionTransformer',
        arch='base',
        img_size=(256, 192),
        patch_size=16,
        qkv_bias=True,
        drop_path_rate=0.3,
        with_cls_token=False,
        out_type='featmap',
        patch_cfg=dict(padding=2),
        init_cfg=dict(
            type='Pretrained',
            checkpoint='https://download.openmmlab.com/mmpose/'
            'v1/pretrained_models/mae_pretrain_vit_base.pth'),
    ),
    head=dict(
        type='HeatmapHead',
        in_channels=768,
        out_channels=17,
        deconv_out_channels=(256, 256),
        deconv_kernel_sizes=(4, 4),
        loss=dict(type='KeypointMSELoss', use_target_weight=True),
        decoder=codec),
    test_cfg=dict(
        flip_test=True,
        flip_mode='heatmap',
        shift_heatmap=False,
    ))

type='mmpretrain.VisionTransformer'
In mmpose, there is not mmpretrain. So I have to download for using this model, but i don't know how to set the module.

This is the part of error messages.

KeyError: 'mmpretrain.VisionTransformer is not in the model registry. Please check whether the value of mmpretrain.VisionTransformer is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'

During handling of the above exception, another exception occurred:

KeyError: "class TopdownPoseEstimator in mmpose/models/pose_estimators/topdown.py: 'mmpretrain.VisionTransformer is not in the model registry. Please check whether the value of mmpretrain.VisionTransformer is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'"

[Ben-Louis]

Please refer to https://github.com/open-mmlab/mmpose/blob/main/configs/body_2d_keypoint/topdown_heatmap/coco/vitpose_coco.md to install mmpretrain

[seon-creator]

Thank you so much!!
It is solved!!

[raahim-syed]

I have been stuck for week procrastinating on this and it is really frustrating to find no solution. I am trying to load the pose estimation model in mmpose and trying to perform inference on it without using the torchserve container found in your repo and I ran into some problems.

I extracted the .mar model file and got a weights file (best_AP_epoch_72.pth) and a config for the model.

channel_cfg = dict(
    num_output_channels=17,
    dataset_joints=17,
    dataset_channel=[[
        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
    ]],
    inference_channel=[
        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
    ])


model = dict(
    type='TopDown',
    pretrained='torchvision://resnet50',
    backbone=dict(type='ResNet', depth=50),
    keypoint_head=dict(
        type='TopdownHeatmapSimpleHead',
        in_channels=2048,
        out_channels=17,
        loss_keypoint=dict(type='JointsMSELoss', use_target_weight=True)),
    train_cfg=dict(),
    test_cfg=dict(
        flip_test=True,
        post_process='default',
        shift_heatmap=True,
        modulate_kernel=11))

When I load this into mmpose for inference I get the following error:

KeyError                                  Traceback (most recent call last)
Cell In[7], line 1
----> 1 model = init_model(CONFIG_PATH, WEIGHTS_PATH)

File /opt/conda/lib/python3.10/site-packages/mmpose/apis/inference.py:104, in init_model(config, checkpoint, device, cfg_options)
    101 if scope is not None:
    102     init_default_scope(scope)
--> 104 model = build_pose_estimator(config.model)
    105 model = revert_sync_batchnorm(model)
    106 # get dataset_meta in this priority: checkpoint > config > default (COCO)

File /opt/conda/lib/python3.10/site-packages/mmpose/models/builder.py:35, in build_pose_estimator(cfg)
     33 def build_pose_estimator(cfg):
     34     """Build pose estimator."""
---> 35     return POSE_ESTIMATORS.build(cfg)

File /opt/conda/lib/python3.10/site-packages/mmengine/registry/registry.py:570, in Registry.build(self, cfg, *args, **kwargs)
    548 def build(self, cfg: dict, *args, **kwargs) -> Any:
    549     """Build an instance.
    550 
    551     Build an instance by calling :attr:`build_func`.
   (...)
    568         >>> model = MODELS.build(cfg)
    569     """
--> 570     return self.build_func(cfg, *args, **kwargs, registry=self)

File /opt/conda/lib/python3.10/site-packages/mmengine/registry/build_functions.py:232, in build_model_from_cfg(cfg, registry, default_args)
    230     return Sequential(*modules)
    231 else:
--> 232     return build_from_cfg(cfg, registry, default_args)

File /opt/conda/lib/python3.10/site-packages/mmengine/registry/build_functions.py:100, in build_from_cfg(cfg, registry, default_args)
     98     obj_cls = registry.get(obj_type)
     99     if obj_cls is None:
--> 100         raise KeyError(
    101             f'{obj_type} is not in the {registry.scope}::{registry.name} registry. '  # noqa: E501
    102             f'Please check whether the value of `{obj_type}` is '
    103             'correct or it was registered as expected. More details '
    104             'can be found at '
    105             '[https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module](https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module%3C/span%3E%3Cspan) style="color:rgb(175,0,0)">'  # noqa: E501
    106         )
    107 # this will include classes, functions, partial functions and more
    108 elif callable(obj_type):

KeyError: 'TopDown is not in the mmpose::model registry. Please check whether the value of `TopDown` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'

The Code I am Using is this:

import time
import torch
from mmpose.apis import init_model, inference_topdown, MMPoseInferencer, Pose2DInferencer
from mmpose.registry import VISUALIZERS
from mmpose.structures import merge_data_samples
from mmdet.apis import inference_detector, init_detector

model = init_model(CONFIG_PATH, WEIGHTS_PATH)

Secondly I tried This:

Since, I had the weights I decided to code the model out in pytorch but its complete architecture was unknown to me especially the TopdownHeatmapSimpleHead but while doing so I wrote this code:

# Example instantiation of the model with the data configuration parameters
model = KeypointModel(
    image_size=(192, 256),  # From data_cfg
    heatmap_size=(48, 64),  # From data_cfg
    num_keypoints=17,       # From data_cfg
    pretrained=True
)


# Path to the weights file
weights_path = 'D:/fyp-codebase/mmpose/weights/best_AP_epoch_72.pth'

# Load the state dictionary from the file
state_dict = torch.load(weights_path, map_location=torch.device('cpu'))

# Load the weights into the model
model.load_state_dict(state_dict)

# Set the model to evaluation mode (if you're using it for inference)
model.eval()

print("Weights loaded successfully!")

And I encountered this error:

Cell In[6], [line 8](vscode-notebook-cell:?execution_count=6&line=8)
      [5](vscode-notebook-cell:?execution_count=6&line=5) state_dict = torch.load(weights_path, map_location=torch.device('cpu'))
      [7](vscode-notebook-cell:?execution_count=6&line=7) # Load the weights into the model
----> [8](vscode-notebook-cell:?execution_count=6&line=8) model.load_state_dict(state_dict)
     [10](vscode-notebook-cell:?execution_count=6&line=10) # Set the model to evaluation mode (if you're using it for inference)
     [11](vscode-notebook-cell:?execution_count=6&line=11) model.eval()

File ~\AppData\Roaming\Python\Python311\site-packages\torch\nn\modules\module.py:2152, in Module.load_state_dict(self, state_dict, strict, assign)
   [2147](https://file+.vscode-resource.vscode-cdn.net/d%3A/fyp-codebase/mmpose/~/AppData/Roaming/Python/Python311/site-packages/torch/nn/modules/module.py:2147)         error_msgs.insert(
   [2148](https://file+.vscode-resource.vscode-cdn.net/d%3A/fyp-codebase/mmpose/~/AppData/Roaming/Python/Python311/site-packages/torch/nn/modules/module.py:2148)             0, 'Missing key(s) in state_dict: {}. '.format(
   [2149](https://file+.vscode-resource.vscode-cdn.net/d%3A/fyp-codebase/mmpose/~/AppData/Roaming/Python/Python311/site-packages/torch/nn/modules/module.py:2149)                 ', '.join(f'"{k}"' for k in missing_keys)))
   [2151](https://file+.vscode-resource.vscode-cdn.net/d%3A/fyp-codebase/mmpose/~/AppData/Roaming/Python/Python311/site-packages/torch/nn/modules/module.py:2151) if len(error_msgs) > 0:
-> [2152](https://file+.vscode-resource.vscode-cdn.net/d%3A/fyp-codebase/mmpose/~/AppData/Roaming/Python/Python311/site-packages/torch/nn/modules/module.py:2152)     raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
   [2153](https://file+.vscode-resource.vscode-cdn.net/d%3A/fyp-codebase/mmpose/~/AppData/Roaming/Python/Python311/site-packages/torch/nn/modules/module.py:2153)                        self.__class__.__name__, "\n\t".join(error_msgs)))
   [2154](https://file+.vscode-resource.vscode-cdn.net/d%3A/fyp-codebase/mmpose/~/AppData/Roaming/Python/Python311/site-packages/torch/nn/modules/module.py:2154) return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for KeypointModel:
	Missing key(s) in state_dict: "backbone.0.weight", "backbone.1.weight", "backbone.1.bias", "backbone.1.running_mean", "backbone.1.running_var", "backbone.4.0.conv1.weight", "backbone.4.0.bn1.weight", "backbone.4.0.bn1.bias", "backbone.4.0.bn1.running_mean", "backbone.4.0.bn1.running_var", "backbone.4.0.conv2.weight", "backbone.4.0.bn2.weight", "backbone.4.0.bn2.bias", "backbone.4.0.bn2.running_mean", "backbone.4.0.bn2.running_var", "backbone.4.0.conv3.weight", "backbone.4.0.bn3.weight", "backbone.4.0.bn3.bias", "backbone.4.0.bn3.running_mean", "backbone.4.0.bn3.running_var", "backbone.4.0.downsample.0.weight", "backbone.4.0.downsample.1.weight", "backbone.4.0.downsample.1.bias", "backbone.4.0.downsample.1.running_mean", "backbone.4.0.downsample.1.running_var", "backbone.4.1.conv1.weight", "backbone.4.1.bn1.weight", "backbone.4.1.bn1.bias", "backbone.4.1.bn1.running_mean", "backbone.4.1.bn1.running_var", "backbone.4.1.conv2.weight", "backbone.4.1.bn2.weight", "backbone.4.1.bn2.bias", "backbone.4.1.bn2.running_mean", "backbone.4.1.bn2.running_var", "backbone.4.1.conv3.weight", "backbone.4.1.bn3.weight", "backbone.4.1.bn3.bias", "backbone.4.1.bn3.running_mean", "backbone.4.1.bn3.running_var", "backbone.4.2.conv1.weight", "backbone.4.2.bn1.weight", "backbone.4.2.bn1.bias", "backbone.4.2.bn1.running_mean", "backbone.4.2.bn1.running_var", "backbone.4.2.conv2.weight", "backbone.4.2.bn2.weight", "backbone.4.2.bn2.bias", "backbone.4.2.bn2.running_mean", "backbone.4.2.bn2.running_var", "backbone.4.2.conv3.weight", "backbone.4.2.bn3.weight", "backbone.4.2.bn3.bias", "backbone.4.2.bn3.running_mean", "backbone.4.2.bn3.running_var", "backbone.5.0.conv1.weight", "backbone.5.0.bn1.weight", "backbone.5.0.bn1.bias", "backbone.5.0.bn1.running_mean", "backbone.5.0.bn1.running_var", "backbone.5.0.conv2.weight", "backbone.5.0.bn2.weight", "backbone.5.0.bn2.bias", "backbone.5.0.bn2.running_mean", "backbone.5.0.bn2.running_var", "backbone.5.0.conv3.weight", "backbone.5.0.bn3.weight", "backbone.5.0.bn3.bias", "backbone.5.0.bn3.running_mean", "backbone.5.0.bn3.running_var", "backbone.5.0.downsample.0.weight", "backbone.5.0.downsample.1.weight", "backbone.5.0.downsample.1.bias", "backbone.5.0.downsample.1.running_mean", "backbone.5.0.downsample.1.running_var", "backbone.5.1.conv1.weight", "backbone.5.1.bn1.weight", "backbone.5.1.bn1.bias", "backbone.5.1.bn1.running_mean", "backbone.5.1.bn1.running_var", "backbone.5.1.conv2.weight", "backbone.5.1.bn2.weight", "backbone.5.1.bn2.bias", "backbone.5.1.bn2.running_mean", "backbone.5.1.bn2.running_var", "backbone.5.1.conv3.weight", "backbone.5.1.bn3.weight", "backbone.5.1.bn3.bias", "backbone.5.1.bn3.running_mean", "backbone.5.1.bn3.running_var", "backbone.5.2.conv1.weight", "backbone.5.2.bn1.weight", "backbone.5.2.bn1.bias", "backbone.5.2.bn1.running_mean", "backbone.5.2.bn1.running_var", "backbone.5.2.conv2.weight", "backbone.5.2.bn2.weight", "backbone.5.2.bn2.bias", "backbone.5.2.bn2.running_mean", "backbone.5.2.bn2.running_var", "backbone.5.2.conv3.weight", "backbone.5.2.bn3.weight", "backbone.5.2.bn3.bias", "backbone.5.2.bn3.running_mean", "backbone.5.2.bn3.running_var", "backbone.5.3.conv1.weight", "backbone.5.3.bn1.weight", "backbone.5.3.bn1.bias", "backbone.5.3.bn1.running_mean", "backbone.5.3.bn1.running_var", "backbone.5.3.conv2.weight", "backbone.5.3.bn2.weight", "backbone.5.3.bn2.bias", "backbone.5.3.bn2.running_mean", "backbone.5.3.bn2.running_var", "backbone.5.3.conv3.weight", "backbone.5.3.bn3.weight", "backbone.5.3.bn3.bias", "backbone.5.3.bn3.running_mean", "backbone.5.3.bn3.running_var", "backbone.6.0.conv1.weight", "backbone.6.0.bn1.weight", "backbone.6.0.bn1.bias", "backbone.6.0.bn1.running_mean", "backbone.6.0.bn1.running_var", "backbone.6.0.conv2.weight", "backbone.6.0.bn2.weight", "backbone.6.0.bn2.bias", "backbone.6.0.bn2.running_mean", "backbone.6.0.bn2.running_var", "backbone.6.0.conv3.weight", "backbone.6.0.bn3.weight", "backbone.6.0.bn3.bias", "backbone.6.0.bn3.running_mean", "backbone.6.0.bn3.running_var", "backbone.6.0.downsample.0.weight", "backbone.6.0.downsample.1.weight", "backbone.6.0.downsample.1.bias", "backbone.6.0.downsample.1.running_mean", "backbone.6.0.downsample.1.running_var", "backbone.6.1.conv1.weight", "backbone.6.1.bn1.weight", "backbone.6.1.bn1.bias", "backbone.6.1.bn1.running_mean", "backbone.6.1.bn1.running_var", "backbone.6.1.conv2.weight", "backbone.6.1.bn2.weight", "backbone.6.1.bn2.bias", "backbone.6.1.bn2.running_mean", "backbone.6.1.bn2.running_var", "backbone.6.1.conv3.weight", "backbone.6.1.bn3.weight", "backbone.6.1.bn3.bias", "backbone.6.1.bn3.running_mean", "backbone.6.1.bn3.running_var", "backbone.6.2.conv1.weight", "backbone.6.2.bn1.weight", "backbone.6.2.bn1.bias", "backbone.6.2.bn1.running_mean", "backbone.6.2.bn1.running_var", "backbone.6.2.conv2.weight", "backbone.6.2.bn2.weight", "backbone.6.2.bn2.bias", "backbone.6.2.bn2.running_mean", "backbone.6.2.bn2.running_var", "backbone.6.2.conv3.weight", "backbone.6.2.bn3.weight", "backbone.6.2.bn3.bias", "backbone.6.2.bn3.running_mean", "backbone.6.2.bn3.running_var", "backbone.6.3.conv1.weight", "backbone.6.3.bn1.weight", "backbone.6.3.bn1.bias", "backbone.6.3.bn1.running_mean", "backbone.6.3.bn1.running_var", "backbone.6.3.conv2.weight", "backbone.6.3.bn2.weight", "backbone.6.3.bn2.bias", "backbone.6.3.bn2.running_mean", "backbone.6.3.bn2.running_var", "backbone.6.3.conv3.weight", "backbone.6.3.bn3.weight", "backbone.6.3.bn3.bias", "backbone.6.3.bn3.running_mean", "backbone.6.3.bn3.running_var", "backbone.6.4.conv1.weight", "backbone.6.4.bn1.weight", "backbone.6.4.bn1.bias", "backbone.6.4.bn1.running_mean", "backbone.6.4.bn1.running_var", "backbone.6.4.conv2.weight", "backbone.6.4.bn2.weight", "backbone.6.4.bn2.bias", "backbone.6.4.bn2.running_mean", "backbone.6.4.bn2.running_var", "backbone.6.4.conv3.weight", "backbone.6.4.bn3.weight", "backbone.6.4.bn3.bias", "backbone.6.4.bn3.running_mean", "backbone.6.4.bn3.running_var", "backbone.6.5.conv1.weight", "backbone.6.5.bn1.weight", "backbone.6.5.bn1.bias", "backbone.6.5.bn1.running_mean", "backbone.6.5.bn1.running_var", "backbone.6.5.conv2.weight", "backbone.6.5.bn2.weight", "backbone.6.5.bn2.bias", "backbone.6.5.bn2.running_mean", "backbone.6.5.bn2.running_var", "backbone.6.5.conv3.weight", "backbone.6.5.bn3.weight", "backbone.6.5.bn3.bias", "backbone.6.5.bn3.running_mean", "backbone.6.5.bn3.running_var", "backbone.7.0.conv1.weight", "backbone.7.0.bn1.weight", "backbone.7.0.bn1.bias", "backbone.7.0.bn1.running_mean", "backbone.7.0.bn1.running_var", "backbone.7.0.conv2.weight", "backbone.7.0.bn2.weight", "backbone.7.0.bn2.bias", "backbone.7.0.bn2.running_mean", "backbone.7.0.bn2.running_var", "backbone.7.0.conv3.weight", "backbone.7.0.bn3.weight", "backbone.7.0.bn3.bias", "backbone.7.0.bn3.running_mean", "backbone.7.0.bn3.running_var", "backbone.7.0.downsample.0.weight", "backbone.7.0.downsample.1.weight", "backbone.7.0.downsample.1.bias", "backbone.7.0.downsample.1.running_mean", "backbone.7.0.downsample.1.running_var", "backbone.7.1.conv1.weight", "backbone.7.1.bn1.weight", "backbone.7.1.bn1.bias", "backbone.7.1.bn1.running_mean", "backbone.7.1.bn1.running_var", "backbone.7.1.conv2.weight", "backbone.7.1.bn2.weight", "backbone.7.1.bn2.bias", "backbone.7.1.bn2.running_mean", "backbone.7.1.bn2.running_var", "backbone.7.1.conv3.weight", "backbone.7.1.bn3.weight", "backbone.7.1.bn3.bias", "backbone.7.1.bn3.running_mean", "backbone.7.1.bn3.running_var", "backbone.7.2.conv1.weight", "backbone.7.2.bn1.weight", "backbone.7.2.bn1.bias", "backbone.7.2.bn1.running_mean", "backbone.7.2.bn1.running_var", "backbone.7.2.conv2.weight", "backbone.7.2.bn2.weight", "backbone.7.2.bn2.bias", "backbone.7.2.bn2.running_mean", "backbone.7.2.bn2.running_var", "backbone.7.2.conv3.weight", "backbone.7.2.bn3.weight", "backbone.7.2.bn3.bias", "backbone.7.2.bn3.running_mean", "backbone.7.2.bn3.running_var", "keypoint_head.weight", "keypoint_head.bias". 
	Unexpected key(s) in state_dict: "meta", "state_dict", "optimizer".

Thank you,
Raahim