dotnet test running assemblies in parallel (when mixing core and framework test assemblies)

[tznind]

I have an issue where "dotnet test" is running assemblies in parallel. I have tried adding [NonParallelizable] and [assembly: LevelOfParallelism(1)] and also running with the dotnet test CLI option -p:ParallelizeTestCollections=false none of which has helped.

I have:

Solution
\ MyLibrary.csproj (netstandard2.0)
\ Tests461 (net461)
\ TestsCore2 (netcoreapp2.2)

The nuget dependencies of both test csproj files are:

<ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.2.0" />
    <PackageReference Include="NUnit" Version="3.12.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.15.1" />
    <PackageReference Include="NunitXml.TestLogger" Version="2.1.41" />
  </ItemGroup>

When I run the tests using

dotnet test

I get the following logged:

Core SetUp 688503812
Core Enter Test 688503812
461 SetUp 688506609
461 Enter Test 688506609
Core Exit Test 688506812
461 Exit Test 688509625

It even looks like dotnet test is firing up 2 instances of the tester process (which don't respect each other's parallelization):

PS E:\MyLibrary> dotnet test
Test run for E:\MyLibrary\TestsCore2\bin\Debug\netcoreapp2.2\TestsCore2.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0-preview-20190715-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run for E:\MyLibrary\Tests461\bin\Debug\net461\Tests461.dll(.NETFramework,Version=v4.6.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0-preview-20190715-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 4.0100 Seconds

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 4.0288 Seconds
PS E:\MyLibrary>

I have built a simple 3 project solution that reproduces the problem if it would help I can provide.

Other things I have tried:

dotnet test -- NUnit.NumberOfTestWorkers=1

[mikkelbu]

Hi @tznind
I've moved this issue to the adapter project, as the dotnet test is executing the tests via the adapter. That being said I don't think this is an issue in NUnit, as the parallelisation of assemblies happens outside of NUnit as far as I know. But I'm unsure of whether this is done by msbuild or vstest, see e.g. microsoft/vstest#1966 (comment). I'm sure that @OsirisTerje knows more 😄.

Ps. I don't think that you can pass options to the underlying adapter, as far as I know you should use a runsettings file, but note that NumberOfTestWorkers would only have affect the parallelization within an assembly.

[OsirisTerje]

@tznind : As @mikkelbu says: Add a runsettings file, and turn off VSTest parallelization there.
Enable it by using the -s <filenameOfRunsettingsfile option:

<?xml version="1.0" encoding="utf-8"?>
<!-- For more information, see https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2019  -->

<RunSettings>
 <RunConfiguration>
   <!-- 0 = As many processes as possible, limited by number of cores on machine, 1 = Sequential (1 process), 2-> Given number of processes up to limit by number of cores on machine-->
   <MaxCpuCount>1</MaxCpuCount>
   <!-- Disables in-assembly parallel execution, applies to both MSTest and NUnit -->
   <DisableParallelization>true</DisableParallelization>
 </RunConfiguration>
</RunSettings>

[tznind]

Hey, thanks for the advice. I have added your settings file and run:

dotnet test -s test.runsettings

I get the same parallel execution (both assemblies are tested simultaneously). It is definetly reading the runsettings file because if I change the path to something that doesn't exist it complains (as it does if the XML is invalid).

It seems like this is not something that can be controlled currently through dotnet test:

microsoft/vstest#1966
https://github.com/dotnet/cli/issues/11017

I have an interim solution which is to just call dotnet test sequentially against each project in my build script with --no-build (though it lacks elegance).

[OsirisTerje]

@tznind Do you have a small repo with this repro'ed you could point to (or upload) ?

[tznind]

Hey sorry for the late reply. I have uploaded the minimum repro:

https://github.com/tznind/DotnetTestParallelExample

[mikeharder]

@tznind: See a workaround I posted here: https://github.com/dotnet/cli/issues/11017#issuecomment-564311592. I verified this works in your minimum repro:

$ dotnet test dirs.proj -p:BuildInParallel=false
Test run for D:\Git\DotnetTestParallelExample\Tests461\bin\Debug\net461\Tests461.dll(.NETFramework,Version=v4.6.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 4.3995 Seconds
Test run for D:\Git\DotnetTestParallelExample\TestsCore2\bin\Debug\netcoreapp2.2\TestsCore2.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 4.7372 Seconds
 
$ type C:\temp\output.txt
461 SetUp -1944219765
461 Enter Test -1944219750
461 Exit Test -1944216750
Core SetUp -1944214390
Core Enter Test -1944214375
Core Exit Test -1944211375

[papatelst]

MaxCpuCount runconfiguration in .runsettings file is ignored by dotnet.exe test and vstest.console.exe commands. see details at -
microsoft/vstest#1966 (comment)
Please help to fix or tell me what am I missing here.
Thanks

[tznind]

MaxCpuCount runconfiguration in .runsettings file is ignored by dotnet.exe test and vstest.console.exe commands. see details at -
microsoft/vstest#1966 (comment)
Please help to fix or tell me what am I missing here.
Thanks

This ticket is about multiple csproj test files running in parallel despite the all the mechanisms designed to stop this happening (like MaxCpuCount - which as you say doesn't work for multiprocess).

There are basically two ways to make multi csproj dotnet test run in sequence (as opposed to parallel).

1. Run from command line with seperate call to each csproj e.g. dotnet test myproj1.csproj then dotnet test myproj2.csproj
2. Create a .proj file (which is like a budget .sln file) and running it with dotnet test -p:BuildInParallel=false dirs.proj (see the link by @mikeharder - https://github.com/dotnet/cli/issues/11017#issuecomment-564311592)

Be aware that once you have a .proj and a .sln in the same directory dotnet can get confused when you enter dotnet build (it will say it's unclear what to build since they are both viable build targets).

Hopefully at some point the BuildInParallel option will work with .sln files

[smchan514]

For anyone still looking for a solution, there is the command line option -m:1 which specifies the maximum number of concurrent processes to use when building.

dotnet test -m:1 MyUnitTests.sln

Ref:

1. Add parameter to ensure tests run in sequence regardless of when build completes dotnet/sdk#10178
2. MSBuild command-line reference -- Switches

[OsirisTerje]

Thanks @smchan514 for that option! That's probably the best one can do.

I'll convert this to a discussion.