Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can Test Explorer match the Expecto folder hierarchy? #431

Open
charlesroddie opened this issue Apr 23, 2022 · 6 comments
Open

Can Test Explorer match the Expecto folder hierarchy? #431

charlesroddie opened this issue Apr 23, 2022 · 6 comments

Comments

@charlesroddie
Copy link

charlesroddie commented Apr 23, 2022

Given tests in a folder structure:

[<Tests>]
let tests = 
  testList "ListA" [
    testList "ListAA" [
      testCase "CaseAA1" id
      testCase "CaseAA2" id
    ]
    testCase "CaseAB1" id
  ]

Is there a way to have Test Explorer match this?

Project, Namespace, Class isn't right
image

Project, Class isn't right either
image

@cmeeren in #364 was getting some correspondence, but the samples there are very constrained - every test is exactly 2 levels deep in the hierarchy.

Is there a spec for the Test Explorer behaviour, or should I figure out by trial and error what Expecto structures result in what outputs and then try to make a set of consrained types (FolderLevel1, FolderLevel2) and then try to generate an altered Expecto folder structure to get the desired Text Explorer behaviour?

@charlesroddie
Copy link
Author

@baronfel I'm taking up your suggestion to try Expecto (via https://github.com/YoloDev/YoloDev.Expecto.TestSdk) and it's good so far but I'm stuck at this point.

@charlesroddie
Copy link
Author

charlesroddie commented Apr 25, 2022

Even "Rectangular hierarchy", in which folders of degree 0 contain only tests, and folders of degree n contain folders of degree n-1, and the main degree is 3, doesn't work:

image

There seems to be a big problem connecting Expecto to Test Explorer. Test Explorer may be hacked together with strings but it definitely allows for deeper hierarchies than this.

Am I right in posting here rather than in https://github.com/YoloDev/YoloDev.Expecto.TestSdk ?

@olivercoad
Copy link

Hi.
It looks like you're using Visual Studio.

I don't use Visual Studio very much anymore, but back in the day I used to use adamchester/expecto-adapter. It looks like it's been archived since then (I guess people use YoloDev.Expecto.TestSdk for that now), but I remember getting the nesting in the VS Test Explorer to work fine. See screenshot in adamchester/expecto-adapter#47. You could try seeing if it still works but it might need an older version of VS.

Usually I just use the terminal for running tests now (ftestCase and ftestList are useful) so don't know much about YoloDev.Expecto.TestSdk, but I tried your example in VSCode to see what it's like with Ionide.

The nesting works fine for what you had in your "Rectangular hierarchy".
image

"Some other test" doesn't even show up in the right place though, it should be nested under "List".
Also, it only seems to identify testCases with a function defined in-line (CaseAAA1 shows up but CaseAAB1 doesn't). And of course, tests generated in a for loop don't show up either.

There seems to be an issue with it Ionide or VSCode though. When I try to run tests from the test explorer it just hangs indefinitely showing the spinner. Nothing shows up in any of the VSCode output logs.
image
The TestResults/Ionide.trx file is generated successfully and has all the expected tests and test names, so I suspect if it actually ran properly it might show all the tests and in the correct spots.

The output of dotnet test -t is exactly what I'd expect it to be, as is the Ionide.trx output file, so I'd guess the issue is probably in the interaction between YoloDev.Expecto.TestSdk and Visual Studio. (And the Ionide/VSCode issue is probably unrelated)

The following Tests are available:
    List.Some other test
    List.ListA.ListAA.ListAAA.CaseAAA1
    List.ListA.ListAA.ListAAA.CaseAAA2
    List.ListA.ListAA.ListAAB.CaseAAB1
    List.ListA.ListAA.ListAAB.CaseAAB2
    List.ListA.ListAB.ListABA-generated-list.CaseABA0
    List.ListA.ListAB.ListABA-generated-list.CaseABA1
    List.ListA.ListAB.ListABA-generated-list.CaseABA2
    List.ListA.ListAB.ListABA-generated-list.CaseABA3
    List.ListA.ListAB.ListABA-generated-list.CaseABA4
    List.ListA.ListAB.ListABB.CaseABB1
    List.ListA.ListAB.ListABB.CaseABB2
...

@baronfel
Copy link
Contributor

Note that the VSCode feature is in its infancy, so I'd err on the side of a bug in our test discovery and execution. This issue should focus purely on the discovery and execution of expecto tests via the Yolodev adapter, which provides the bridge to the 'dotnet test' world that powers the CLI and VS experiences.

@olivercoad
Copy link

How does the VSCode feature discover and execute tests then (and where is it implemented?) if it's not with the yolodev adapter? I thought it runs something like dotnet test --logger "trx;Ionide.trx" to generate the report, meaning it would be using the same "'dotnet test' world" as VS.

But agreed, this issue should focus on the Yolodev adapter with VS

@baronfel
Copy link
Contributor

The discovery happens via syntax analysis, which is way faster but not as accurate. The execution should happen via dotnet test, however. Can you raise an issue about any test execution issues on the Ionide VSCode repo?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants