MSVC 2022 crash when dynamic casting

[thomaspeer]

Hi,

We recently wanted to use dynamic casting to check if some collision shape is a btScaledBvhTriangleMeshShape but we are facing a strange crash in VS2022.

Problem description

Below a sample code we managed to consistently reproduce the issue with:

// Create our basic btTriangleMesh
btTriangleMesh* triangleMesh = new btTriangleMesh();
triangleMesh->addTriangle(btVector3(0, 0, 0), btVector3(0, 1, 0), btVector3(1, 1, 0));

// Create our btBvhTriangleMeshShape
btBvhTriangleMeshShape* triangleMeshShape = new btBvhTriangleMeshShape(triangleMesh, true, true);

// Create our btScaledBvhTriangleMeshShape
btScaledBvhTriangleMeshShape* scaledTriangleMeshShape = new btScaledBvhTriangleMeshShape(triangleMeshShape, btVector3(1, 1, 1));

// Cast the btScaledBvhTriangleMeshShapeto a btCollisionShape (works fine)
btCollisionShape* newCollisionShape = dynamic_cast<btCollisionShape*>(scaledTriangleMeshShape);

// Cast the btCollisionShape to a btBvhTriangleMeshShape  (crashes)
btBvhTriangleMeshShape* newTriangleMeshShape = dynamic_cast<btBvhTriangleMeshShape*>(newCollisionShape);

This also happens if doing as follows:

// Cast the btBvhTriangleMeshShapea btCollisionShape (works fine)
btCollisionShape* newCollisionShape = dynamic_cast<btCollisionShape*>(triangleMeshShape riangleMeshShape);

// Cast the btCollisionShape to a btBvhTriangleMeshShape  (crashes)
btBvhTriangleMeshShape* newTriangleMeshShape = dynamic_cast<btBvhTriangleMeshShape*>(newCollisionShape);

The screenshot below shows the resulting crash in the debugger:

Additional findings

I noticed that if I subclass btScaledBvhTriangleMeshShape, the problem vanishes:

class ScaledBvhTriangleMeshShape final : public btScaledBvhTriangleMeshShape
{
public:
    ScaledBvhTriangleMeshShape(btBvhTriangleMeshShape* child_shape, const btVector3& local_scaling)
        : btScaledBvhTriangleMeshShape(child_shape, local_scaling)
    {
    }
};

Maybe this is some alignment issues with MSVC and dynamic cast?

Thanks for helping!

System specs

In case that helps:

• CPU: Intel 13600KF
• GPU: AMD 6950 XT
• RAM: Corsair 64 GB
• Operating system: Windows 11 Professional 22631.3880
• IDE: VS2022 Version 17.11.2
• Build type: Debug