feat: Support embedding of the type library into the assembly.

[bclothier]

In .NET 8, they introduced the ability to embed the type library into the *.comhost.dll file with the ComHostTypeLibrary property. However, this property is not compatible with projects using dscom to help with exporting the type library. The reason for this is because the property expects a type library to be already present at the start of the build whereas dscom's build generates a type library as part of the build process.

With this PR, we gain the ability to embed the type library that was generated as part of the build. This is exposed both as a new switch on the client tool, using --tlbembed option and as a new build task TlbEmbed which is run after the TlbExport build task. This is an opt-in step by using the new property DsComTypeLibraryEmbedAfterBuild which is defaulted to false.

[bclothier]

@marklechtermann or @carstencodes FYI - I put this in draft because there is one step I am having difficulty with validating and that is passing the property DsComTypeLibraryEmbedAfterBuild to the tool build. In the tool target file, we have this:

dscom/src/dscom.build/dSPACE.Runtime.InteropServices.BuildTasks.Tools.targets

Line 52 in fb03884

<_DsComArguments> $(_DsComArguments) --tlbembed</_DsComArguments>

I temporarily omitted the Condition for my local test. I used a local nuget feed to import the prerelease nuget so that I can test with my project to verify this is working from a nuget package. However in my testing, the property never get set, even though my project csproj file has DsComTypeLibraryEmbedAfterBuild set. I thought it might have been a caching and tried to clear nuget cache for the packages but the target still does not set the expected --tblembed when constructing the _DsComArguments. It's possible I'm missing something as I am not fully familiar with authoring custom build targets, but want to first verify the nuget package will work with the new feature. Any suggestions to try?

[marklechtermann]

There is still an error in the build task acceptance-test-msbuild (.github\workflows\example-test.yaml).

You can check this locally by executing the script examples\32bit\scripts\msbuild-acceptance-test.cmd.

[carstencodes]

@bclothier

One general issue I observed was #104 which leads to the issue that TLBs were missing or empty, if dscom was used inside MsBuild environments. @marklechtermann decided, that using the executable was a good workaround.

I was also about to delete that part once I got time.

My current suggestion is not to use dscom related functions hosted in MsBuild but to use the executables.

---

Next thing I would suggest is going more straight forward: Why add the ability to add an arbitrary TLB to the Assembly with the TlbExport Task. If there is the necessity to add a special id, I can provide an msbuild target to find the next free id.

Is this really necessary, as this already works for existing TLBs in MsBuild?

---

You wrote, that it does not work that tlbembed parameter is not set. Can you provide me a .binlog file (using dotnet msbuild -bl) or a verbose / detailed / diagnostic log file that would help us investigating us more?

[bclothier]

@bclothier

One general issue I observed was #104 which leads to the issue that TLBs were missing or empty, if dscom was used inside MsBuild environments. @marklechtermann decided, that using the executable was a good workaround.

FYI the documentation seems to imply that both are in use, that the client tool would be only used if 32-bit target is selected. On the review, and if I understand you correctly, the client tool is currently always used and the build task is never used unless I pass false explicitly to the "private" property _DsComForceToolUsage which the documentation does explicitly say to not do.

I think the documentation should be updated to indicate that client tool is always used. In particular this passage:

dscom/README.md

Lines 241 to 245 in 869f481

	The native build task is automatically selected, if a .NET 4.8 or .NET 6.0 assembly for Windows is being build using an x64 platform.
	#### Using the CLI based task
	The CLI task is automatically selected, if a .NET Standard 2.0 assembly is build. It is also chosen, if the target platform is set to x86.

Next thing I would suggest is going more straight forward: Why add the ability to add an arbitrary TLB to the Assembly with the TlbExport Task. If there is the necessity to add a special id, I can provide an msbuild target to find the next free id.

Is this really necessary, as this already works for existing TLBs in MsBuild?

I'm not sure I understand your concern so I'll try to answer:

If you are referring to the ComHostTypeLibrary property, the problems are as follows:

• As I understand it, it works only for a .NET 6+ project. This feature makes it available for .NET 4.8 and .NET 5.
• Even if we add Condition to prevent build error due to a missing type library, it's necessary to build twice to then embed the generated type library which is an extra step and easy to overlook. This change avoids the need to build twice and allows for embedding type library right away in a single build.

If you are referring to the setting the ID to an arbitrary number when adding the resource, the PR will not allow this; it assumes that the index is always 1, which is sensible for the following reasons:

• This is a newly built assembly which won't have any type library resources to start with.
• While it is technically possible to have multiple type library resources embedded in a single PE file, it is not typical and not all applications are capable of parsing multiple type libraries. By convention, it is expected that a file will have a single type library resource. For this purpose, it's sufficient to have a hard-coded index of 1.

Did I address your concerns?

You wrote, that it does not work that tlbembed parameter is not set. Can you provide me a .binlog file (using dotnet msbuild -bl) or a verbose / detailed / diagnostic log file that would help us investigating us more?

Thanks for the suggestion. I will post once I've done more local testing and if I'm stuck.

[carstencodes]

I added a new issue to remove the BuildTask from the package and the documentation.

As the library writer still may have issues, it's better to use the CLI and hence use a verb, if you want to add any TLBs to the assembly.

As I understand it, it works only for a .NET 6+ project. This feature makes it available for .NET 4.8 and .NET 5.

Ok, I see. I think it's up to @SOsterbrink, @matthiasnissen and @marklechtermann to check, if .NET 5 - as it is EOL - should still be targeted. I have no opinion on that.

Even if we add Condition to prevent build error due to a missing type library, it's necessary to build twice to then embed the generated type library which is an extra step and easy to overlook. This change avoids the need to build twice and allows for embedding type library right away in a single build.

Yeah, ok, I think I could not make my point clear.

If we use --tlbembed parameter during build, the TLB would be embedded as a post-build part using id 1. Addidtional TLBs that would be embedded are handled by MsBuild / .NET SDK 6+, as these TLBs are added during compile time to the resulting assemblies. Now the question is: Is this part of the .NET SDK or also a runtime feature?
Meaning: If I compile for .NET 4.8 or .NET 5 using .NET 6 SDK, this should work as well, as the tasks and targets to be called are part of the SDK.

And please don't get me wrong: I appreciate your work. But I'm taking a closer look at the workflow right now.
I actually see the issue, that you need to build to create the TLB to embed into your assembly. That's why I was asking, whether --tlbemb should be enough.

While it is technically possible to have multiple type library resources embedded in a single PE file, it is not typical and not all applications are capable of parsing multiple type libraries. By convention, it is expected that a file will have a single type library resource. For this purpose, it's sufficient to have a hard-coded index of 1.

Ok, I see. If the issue comes around, we can still add a parameter for that. /cc @marklechtermann

So, in conclusion:

• Is there a need for TlbEmbed Task? If so, we need a (maybe separate) command.
• If the --tlbembed flag is still sufficient for the use case and no other TLB must be embedded after the build, we can just rely on --tlbembed to do its work

Or did I get anything wrong here?

[marklechtermann]

First of all, thank you once again for your commitment! @bclothier

@carstencodes is more familiar with the BuildTask. I trust your assessment ;-)

Regarding .NET 5:
In my opinion, .NET 5 should no longer be considered.

Regarding 32Bit:
However, the DSCOM CLI (exe) is available for both 32-bit and 64-bit.
This means that the CLI can only generate a TLB for 32Bit or for 64Bit.
The library itself depends on the host process, whether 32Bit or 64Bit is relevant.

From my point of view, it would be okay if the "tlbembed" feature were only available in the case of 64Bit.
Personally, I think the 32-bit scenario is negligible.

Regarding build task CLI vs library:

My current suggestion is not to use dscom related functions hosted in MsBuild but to use the executables.

I agree

I think the documentation should be updated to indicate that client tool is always used

I agree

I would suggest that the dscom gets a new CLI command.
Something like:

dscom tlbembed <TypeLibrary> <Assembly>

Since the DLL can be changed, I don't know if assembly signing is a problem.
This would then have to be documented.

The build Task should use the new dscom cli command to embed the TLB into the assembly.

[bclothier]

Replying to both @carstencodes and @marklechtermann :

Ok, I see. I think it's up to @SOsterbrink, @matthiasnissen and @marklechtermann to check, if .NET 5 - as it is EOL - should still be targeted. I have no opinion on that.

Regarding .NET 5: In my opinion, .NET 5 should no longer be considered.

Agreed. It's mainly NET48 that might be more interesting.

If we use --tlbembed parameter during build, the TLB would be embedded as a post-build part using id 1. Addidtional TLBs that would be embedded are handled by MsBuild / .NET SDK 6+, as these TLBs are added during compile time to the resulting assemblies. Now the question is: Is this part of the .NET SDK or also a runtime feature? Meaning: If I compile for .NET 4.8 or .NET 5 using .NET 6 SDK, this should work as well, as the tasks and targets to be called are part of the SDK.

I see what you mean now. I assumed that the feature would not be available if a project wasn't targeting .NET6+. Even if my assumption was incorrect, we still have the fact of building twice after a Clean or from a new setup. That is not conductive for automated build.

• Is there a need for TlbEmbed Task? If so, we need a (maybe separate) command.
• If the --tlbembed flag is still sufficient for the use case and no other TLB must be embedded after the build, we can just rely on --tlbembed to do its work

Since the DLL can be changed, I don't know if assembly signing is a problem. This would then have to be documented.

The build Task should use the new dscom cli command to embed the TLB into the assembly.

Good question! It is possible to expose this as a new command verb which is why I made TypeLibEmbedder a separate class rather than updating the existing TypeLibConverter. However, I did not bother making a new command because unless I misunderstand, the build task can only export a type library from the assembly that's being built.

When using the dscom as a CLI, I can see how it would be useful if you wanted to do arbitrary embedding of various TLBs, but I do not see how that would be done as a part of the build task operating on the assembly being built? Even if we convert to a TaskItems, it would imply that the type library is coming from somewhere else outside the assembly being built in which case it becomes equivalent to the existing ComHostTypeLibrary property and I have to then ask why not use it in the first place?

I can build out a new command to expose tlbembed as a separate verb, in addition to the existing --tlbembed switch which would operate on the exported assembly but I think the ability to perform arbitrary embedding of TLB outside the assembly being built would be a separate PR. Will that be OK?

Regarding 32Bit: However, the DSCOM CLI (exe) is available for both 32-bit and 64-bit. This means that the CLI can only generate a TLB for 32Bit or for 64Bit. The library itself depends on the host process, whether 32Bit or 64Bit is relevant.

From my point of view, it would be okay if the "tlbembed" feature were only available in the case of 64Bit. Personally, I think the 32-bit scenario is negligible.

In my scenario, I intend to cross compile and generate a both 32-bit and 64-bit DLL to be then shipped and therefore expect the ability to embed a 32-bit TLB into a 32-bit DLL and 64-bit TLB into a 64-bit DLL.

[marklechtermann]

I can build out a new command to expose tlbembed as a separate verb, in addition to the existing --tlbembed switch which would operate on the exported assembly but I think the ability to perform arbitrary embedding of TLB outside the assembly being built would be a separate PR. Will that be OK?

Sound good to me!

[carstencodes]

@bclothier

Sorry, I've been on sick leave for five days. I'm trying to catch up.

Even if my assumption was incorrect, we still have the fact of building twice after a Clean or from a new setup. That is not conductive for automated build.

How is that possible? The TLB is generated during build and then embedded into the DLL via --tlbembed. What do I miss? So it would be one build to get the desired result?

However, I did not bother making a new command because unless I misunderstand, the build task can only export a type library from the assembly that's being built.

Correct.

Even if we convert to a TaskItems, it would imply that the type library is coming from somewhere else outside the assembly being built in which case it becomes equivalent to the existing ComHostTypeLibrary property and I have to then ask why not use it in the first place?

Exactly. If we put these pieces together, in addition to #280 , I think, we can skip the TlbEmbed task. Would you mind, if I remove the TlbEmbed task with the solution of #280?

I can build out a new command to expose tlbembed as a separate verb, in addition to the existing --tlbembed switch which would operate on the exported assembly but I think the ability to perform arbitrary embedding of TLB outside the assembly being built would be a separate PR

Sounds good to me.

Since the DLL can be changed, I don't know if assembly signing is a problem.
This would then have to be documented.

I think this point @marklechtermann showed up, should be definitly mentioned. This also applies to Strong Names that still have a purpose in .NET 4.8. So tlbembed must be denied, if the assembly has a strong name or we must be sure that dscom is called before the Strong name is applied. Or am I wrong, @marklechtermann ?

So in conclusion: As soon as the failing acceptance test is fixed, I think we can start merging it.

[bclothier]

Sorry, I've been on sick leave for five days. I'm trying to catch up.

I hope you're feeling better!

How is that possible? The TLB is generated during build and then embedded into the DLL via --tlbembed. What do I miss? So it would be one build to get the desired result?

Sorry, that comment was in reference to what would happen if one used ComHostTypeLibrary for such project. You are correct that is the exact purpose that the --tlbembed changes in this PR tries to solve.

Exactly. If we put these pieces together, in addition to #280 , I think, we can skip the TlbEmbed task. Would you mind, if I remove the TlbEmbed task with the solution of #280?

Yes, agreed. I now understand what was the problem and it was indeed with the build task. The tool task are working fine. I will not enhance the build task any further. However, I still have some work to do on the tool task & unit tests & acceptance tests to address the new changes.

I think this point @marklechtermann showed up, should be definitly mentioned. This also applies to Strong Names that still have a purpose in .NET 4.8. So tlbembed must be denied, if the assembly has a strong name or we must be sure that dscom is called before the Strong name is applied. Or am I wrong, @marklechtermann ?

Yes, I agree. I'm not too sure when this will happen but if for some reason it happens before the tool task runs, that would be a problem. For now, I'll update the documentation to make it clear that this is not a tested/supported scenario and they might need to consider using a build script outside the msbuild.

So in conclusion: As soon as the failing acceptance test is fixed, I think we can start merging it.

It is now passing locally but as mentioned, I still have more work, and hope to have an update later this week.

[bclothier]

So according to the log, this is where it's failing, on line 47 when executing the constructor for the CLITestEmbed class.

dscom/src/dscom.test/tests/CLITest/CLITest.cs

Lines 45 to 48 in 9f46ae6

	foreach (var file in Directory.EnumerateFiles(Environment.CurrentDirectory, "*.tlb"))
	{
	File.Delete(file);
	}

I just ran dotnet test --no-build -c Release locally after doing a solution clean and it seems to work fine. Do I need to adjust the expectation here?

[marklechtermann]

This is really strange!
I can't reproduce the error myself either.
I also set up a Windows 2022 server in Azure and tried the github workflow locally with nektos/act.
It works here too.
I can only imagine that this is a timing problem. Maybe the tlb is being blocked by a virus scanner.

Can you implement some kind of retry?
I know that's really ugly!

Something like this:

int retryAttempts = 5;
while (retryAttempts > 0)
{
    try
    {
        foreach (var file in Directory.EnumerateFiles(Environment.CurrentDirectory, "*.tlb"))
        {
            File.Delete(file);
        }
    }
    catch (System.UnauthorizedAccessException ex)
    {
        Thread.Sleep(500);
    } 
    finally 
    {
      retryAttempts--;
   }
}

[carstencodes]

At least on NUnit there is a way to set retries by the framework using a special attribute. But I don't know if the feature is available in XUnit or if this is considered as bad practice.

[bclothier]

Technically the tests in the CLITests and CLITestEmbeds comes more closer to integration tests than true unit tests since they involve working with a built CLI tool. The embed tests makes it worse because it has to run two CLI commands, first to built a TLB to be then used for the testing.

We'll see if the retry logic helps.

[marklechtermann]

...failed. Any other idea?

[bclothier]

So it's not a timing issue. I wonder if running those tests on a STA thread would help avoid issues. Going to try this theory out.

[marklechtermann]

.. finally. This was the solution. Thank you for your work!

[marklechtermann]

New version is v1.11.0:
https://github.com/dspace-group/dscom/releases/tag/v1.11.0
The nuget package will be available in a few minutes.