Migration from GCC to Clang

[BCSharp]

I'm starting a new topic here to keep findings and discussion related to the migration to Clang separate from the general discussion on Ironclad 3.

The major benefit from using Clang is that it can produce .pdb files, while GCC cannot. Also using Clang rather than MSVC should make porting to other systems easier. For Windows, however, I use clang-cl, which is a clang driver providing compatibility with cl (MSVC compiler). Also, for linking I use lld which replaces ld, GCC linker. Here too, the actual linker command is lld-link, which provides compatibility with link (MSVC linker). Using MSVC directly is not feasible because the 64-bit MSVC v.1600 (aka 10.0, or VS 2010, used to compile CPython 3.4) is not freely available. Clang, however, can impersonate MSVC v.1600. I'm using Clang version 16.0.5.

The current status is that both ic_msvcr90.dll and python34.dll are being built with Clang and have corresponding PDB files produced. Tested with VS2022 (ver. 17.7.6): breakpoints in native code work, also stepping through from managed to native works fine. The migration work is done on branch ipy3-clang, which I try to keep rebased on main, until it is stable, proven, and ready to be merged into main. So far so good, so in the coming time I intend to have all Ironclad's native code built with Clang. During the migration I have encountered three significant differences between GCC and Clang that affect the project.

1. GCC (or rather: ld) can link with "naked" DLLs, Clang (lld-link), like link, requires an import library.
2. GCC can automatically export all symbols when building a DLL. Clang-cl, like MSVC, requires either a .def file or a dllexport declaration on exported functions.
3. ld can ignore duplicate data definitions, discarding all but the first encountered. lld-link, like link requires exactly one.

Pro memoria, I will address all these points in more detail in separate posts.

[BCSharp]

Linking with msvcr100.dll

CPython 3.4 uses msvcr100.dll as the C runtime library. Consequently, this is the library that Ironclad 3.4 has to use. Some CPython's installations on Windows (like the one in the Dockerfile) have this library in the same directory as python.exe. However, this is an older revision of the library; the one in the Docker container is 10.00.40219.1. Microsoft still provides a redistributable for 10.0, which contains an updated revision. The one in the container is 10.00.40219.325. Presumably this is the last revision, since the 10.0 version is past end-of-support.

PS C:\> (Get-Item C:\Python34\msvcr100.dll, C:\Windows\System32\msvcr100.dll).VersionInfo

ProductVersion   FileVersion      FileName
--------------   -----------      --------
10.00.40219.1    10.00.40219.1    C:\Python34\msvcr100.dll
10.00.40219.325  10.00.40219.325  C:\Windows\System32\msvcr100.dll

The redistributable is provided to facilitate running applications that already require this version (like CPython 3.4), not to develop new applications or modules (like Ironclad 3.4). Therefore it does not contain the import library (.lib) necessary to link new DLLs to msvcr100.dll. When using the GCC compiler, it seems to be sufficient to add the DLL to the list of object to be linked with ld. The Microsoft toolchain insists on having an implib. The same applies for MSVC-compatible Clang toolchain.

Luckily, the MinGW64 tools collection in the Docker container has both gendef, a tool to extract a .def file from a .dll, and dlltool, to generate an implib from a .def file without any header files. I suppose that pexports would also work in place of gendef. pexports is already used elsewhere during the build of Ironclad. Perhaps even Microsoft dumpbin /exports plus some parsing scripts would do. I do not know of any alternative to dlltool though. Interestingly, SConstruct already contains some provisions for this tool; it used to generate a patched implib for msvcr90.dll for Ironclad 2.6. The build process was replacing libmsvcr90.a that came with MinGW with a version generated using pexports/dlltool. So the fix for Clang uses something that goes back to what Ironclad had already been doing in the past (more or less).

When Ironclad starts supporting ipy3.6, it will switch over to msvcr140.dll, for which the implib is freely available, so this whole concoction will hopefully become obsolete.

[BCSharp]

I saw that when I was trying to use MSVC for compilation. It requires Win7 SDK installed, and I could not find a freely-available official download of it that could be installed automatically inside a docker container. Nevertheless, when I tested it out with old install I was pleased to see that the Service Pack comes with the 64bit compiler, though now with Clang this is "too little too late" (there is even no vcvars64.bat).

If we link with msvcrt.lib rather than msvcr100.lib, woudn't that mean linking with msvcrt.dll? Unless you run on Windows 7, msvcrt.dll would be different than msvcr100.dll, I think... 🤔

However I am now less concerned about msvcrt.lib (gendef/dlltool does a pretty good job), and more about the header files. Currently Clang is simply using the header files from 14.37.32822, not 10.0. It works surprisingly well (to my surprise, at least). I suppose, the C interface didn't change much, if at all. Maybe with C++ it would have been a different case. The biggest incompatibility was with the preprocessor defined macros. You would think that defining __MSVCRT_VERSION__=0x1000 should keep them compatible with 10.0 but it is not quite so. Perhaps 10.0 support was purged over the years. I was able to handle whatever I encountered with some hacks I put in stubmain.c, but using proper header files would be better.

So I'm not sure it is worth the effort to try to assemble a Docker image with all these tools, but if you feel inspired and have an idea how to do it, it would be worth the try to use those header files.

[slozier]

I guess I should have linked the SDK as well. https://www.microsoft.com/en-ca/download/details.aspx?id=8279.

Based on my understanding it's called msvcrt.lib for all versions of the compiler but the DLL varies based on the version. So in this case the DLL would be msvcr100.dll. Seems like there are some header files available as well.

Anyway, if it all works then maybe not worth the effort to package things up at the moment.

[BCSharp]

Thanks for the explanation. The SDK installs OK in the sandbox though it complains that .NET 4 is not installed. No matter what .NET 4 version I install, it still complains. Functionally it's not an issue I guess since the VC compiler KB update installs the missing compiler support anyway without complaining, but I hope there is a way to install the SDK unattended without complaining (in a Docker container).

The libs and header files seem to be complete, so I may have a look at this again in the near future. For now, the current setup works sufficiently well that I would like to merge it with main and move on to other areas of development. Let me know if you have and further comments about my ipy3-clang branch, and if not, I will ff-merge main with it.

[slozier]

Go ahead with the merge, looks good to me.

[BCSharp]

I have found an ISO file download containing Windows 7 SDK, suitable to be installed in the Docker container. It is not a clean install though. It detects .NET Framework 4.8.1 pre-installed and complains that this is a pre-release of 4.0. However it is not possible to install 4.0 because "a newer framework is already present". As a result, some parts of the SDK are not installed. Installing the "Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1" on top fills in some of the holes, but not all. Luckily, enough for our needs.

I have updated the build script and source code to work with MSVC v16.0. It comes with its own set of quirks, but it works. Since it depends on components that are already out-of-support and buggy, I have retained the Clang-based build as well. For the same reason the Clang-based build is the default. To perform an MSVC-based build, add use_msvc1600=yes to the scons command line or to the OS environment variable SCONSFLAGS. scons --clean beforehand is prudent.

I also tried to use Clang with the header files that come with MSVCRT 10.0. These include Windows header files that come with Windows 7 SDK and Clang cannot handle those. Literally, it cannot compile them. So I gave up on that path.

[BCSharp]

Exporting symbols form DLL

GCC runs with a command-line switch to exports all symbols in a DLL that are exported by all individual C modules used to build the DLL. Clang and MSVC require the DLL-exported symbols be specified in a DEF file or labeled with attribute dllexport in the source code. There is also a linker command-line option, but not practical, since it requires each symbol to be specified individually on the command line. Imagine that! The LLD documentation mentions flag -export-all-symbols, which is accepted by lld without a warning (meaning it is being recognized by the linker), but appears to have no effect.

The python34.dll stub is being built using original CPython header files, which are already prepared for MSVC (though not quite for Clang-cl). Using the preprocessor, every public API symbol will have attribute dllexport or dllimport applied, if symbol HAVE_DECLSPEC_DLL is defined. Whether it is dllexport or dllimport depends on whether Py_BUILD_CORE is or is not defined, respectively (logic in pyport.h). There is also no need to define HAVE_DECLSPEC_DLL on command-line; it is defined in PC/pyconfig.h, which is included by all Windows builds.

However, this only applies to functions/data actually defined in the stubs that include those declarations. It does not apply to extra functions/data that are defined in the stub code, nor to the jumptable symbols ("global" symbols in the assembly code). In the end, I ended up by reusing the python34.def that was already being generated early during the build (as input for code generators), plus some explicit dllexport attributes applied to the extras. A nice feature of Clang is #pragma clang attribute push/pop, which reduces the number of changes required. Example:

#pragma clang attribute push (__declspec(dllexport), apply_to=function)

// exported functions...

#pragma clang attribute pop

Other compilers than Clang are obliged to silently ignore #pragma clang.

[BCSharp]

Duplicate data definitions

The GCC is configured to allow duplicate data definitions in different modules through the command-line switch -fcommon. I find it non-C-intuitive, and apparently it may have performance impact. I haven't found an equivalent switch in clang-cl. The Clang documentation lists --allow-multiple-definition, but it is not being accepted by clang-cl. There is a comment in SConstruct that reads # TODO: get rid of -fcommon. So the transition to Clang is the time to get rid of it.

There are two places where Ironclad uses duplicate data definitions. One is in the generated header file _extra_functions.generated.h. Some of those functions are generated as function declarations (prototypes), but some are defined as function pointers. This case is simple: there should be no data definitions in a header file, so I changed the definitions to declarations by slapping the extern modifier on everything. This is redundant on function prototypes, but it is simple and doesn't harm.

The other place is ironclad-data.c. This is a hand-crafted module that contains necessary data definitions that do not come with the stub snippets (i.e. stub/*/*.c). As more CPython code is added to to the snippets, it may contain the definitions of some of those data. So far there is only six types that have duplicate definitions. I have marked them as extern rather than deleting them completely, since the stub code is in a flux and the definitions in the stubs may disappear again. However, keeping them in ironclad-data.c means that the mapper will initialize them (PythonMapper_regiter_types.Generated.cs), and they are already initialized in the C stub code. I'm not sure whether it is simply redundant (in which case the mapper generator can be instructed to skip over extern declarations), or creates some incompatibility. Perhaps investigating what is happening in case of PyUnicode_Type is a good starting point.

[slozier]

It looks like clang-cl accepts the -fcommon option, but I'm fine with disabling.

Looks like most of the ones you marked as extern are likely to remain as C implementations and are listed at the top of _pure_c_symbols. Although, PyUnicode_Type is probably worth looking into. My current plan for unicode it to deal with it as if it were a opaque struct and only deal with it via Python APIs. We probably don't even need to have PyUnicode_Type appear in managed code. I need to look into this some more because it appears to be getting called (incorrectly) by StoreTyped(PythonType)...

[BCSharp]

It looks like clang-cl accepts the -fcommon option, but I'm fine with disabling.

Indeed! Funny that I haven't tested the most obvious case. Good to know for the future. I guess Clang wants to be a drop-in replacement, so aspires to accept everything GCC does.

[BCSharp]

Looks like most of the ones you marked as extern are likely to remain as C implementations and are listed at the top of _pure_c_symbols.

Once the CLang branch is merged with main, I'll leave up to you to remove the superfluous extern as you continue to work on the stub.

PyUnicode_Type is probably worth looking into. My current plan for unicode it to deal with it as if it were a opaque struct and only deal with it via Python APIs.

I may be completely clueless here, but isn't PyUnicode_Type the same as str? Consequently, wouldn't it be mapped to System.String, or at least ExtensibleString rather than being an opaque struct?

Anyway, I'll leave it up to you. To avoid duplication of effort, perhaps I should focus on something different then, like the buffer protocol...

[slozier]

I may be completely clueless here, but isn't PyUnicode_Type the same as str?

Yeah it is, I guess I was talking more about the instance objects that should probably be marshalled as opaque structs and not the type itself (though I see no reason to have a managed definition for it).