"Invalid type found in ExtractAllFixtures for test suite" exception with derived TestFixtureType in IFixtureBuilder

[loop-evgeny]

• NUnit 3.13.2, NUnit3TestAdapter 4.2.1
• Visual Studio 2022 Pro 17.6.7 (but reproducible with dotnet test)
• .NET 6.0, SDK 8.0 RC1 (but happened with SDK 6.0 and 7.0 too)

I have a custom IFixtureBuilder like this

    [AttributeUsage(AttributeTargets.Class)]
    public class EngineTestFixtureAttribute : NUnitAttribute, IFixtureBuilder
    {
        private class OrderedTestFixture : TestFixture
        {
            public OrderedTestFixture(ITypeInfo fixtureType) : base(fixtureType)
            {
                MaintainTestOrder = true;
            }
        }

        public IEnumerable<TestSuite> BuildFrom(ITypeInfo typeInfo)
        {
...
            var fixture = new OrderedTestFixture(typeInfo);
...
            return new[] {fixture};
        }
    }

When run with dotnet test this fails immediately with the following exception:

Microsoft (R) Test Execution Command Line Tool Version 17.8.0-preview-23421-06 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Exception NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryException,    Exception thrown executing tests in MYTESTS.dll
Invalid type found in ExtractAllFixtures for test suite: OrderedTestFixture
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 217
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 204
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 204
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 204
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 204
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractTestFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 273
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 235
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.ConvertXml(NUnitResults discovery) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 192
   at NUnit.VisualStudio.TestAdapter.NUnitEngine.DiscoveryConverter.Convert(NUnitResults discoveryResults, String assemblyPath) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\DiscoveryConverter.cs:line 146
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, IGrouping`2 testCases, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 286
No test is available in MYTESTS.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

The same code worked with NUnit3TestAdapter 3.17.0 (NUnit 3.12.0).

[loop-evgeny]

Looking at the code in

nunit3-vs-adapter/src/NUnitTestAdapter/NUnitEngine/DiscoveryConverter.cs

Line 202 in f3b5c9c

private void ExtractAllFixtures(NUnitDiscoveryCanHaveTestFixture parent, XElement node)

it indeed checks for concrete TestFixture types and throws for derived types.

[loop-evgeny]

In this case I can work around the issue by creating a normal TestFixture and setting MaintainTestOrder = true using Reflection, but that is, of course a hack, and NUnitTestAdapter should really support derived types as it used to do.

[OsirisTerje]

@loop-evgeny Since you find the place where it happened, could you add a test and a fix for the issue as a PR ?

[loop-evgeny]

Maybe! I'd like to, but now that we have a workaround, I'm not sure when I will get around to it.

[loop-evgeny]

@OsirisTerje I debugged this a bit, and it seems that it's not that simple. The discovery XML contains <test-suite type="OrderedTestFixture" ... i.e. only my custom type name - we don't know the base type at that point.

I then checked out the old 3.17 version to see how it worked and it takes a completely different code path in Discovery.cs with settings.DiscoveryMethod == DiscoveryMethod.Old. Seems like it relies on NUnitResults.TestCases() to extract the list of test cases, which simply does return TopNode.SelectNodes("//test-case");

So ExtractAllFixtures probably could be made to "work" for this case by treating any unknown type as a TestFixture, but then what if someone else inherits from SetUpFixture, etc.?

Maybe the proper fix would be to write the recognized base type to the discovery XML (and maybe the real type, too, if that's needed for something else), but I'm out of my depth there and would need to get much deeper into the project than I'm likely to have time for any time soon.