test: mark test-worker-prof as flaky on smartos

[joyeecheung]

This has been flaky at least since Dec 17 and has failed 7 PRs in the last 100 CI runs. I don't think it would do us any good to keep it flaking on smartos in the CI, and I don't think anyone is looking into it, so mark it as flaky.

cc @nodejs/platform-smartos @nodejs/tsc this is what I mentioned in nodejs/TSC#1666 about the tier 2 platforms destablizing the CI. It's counting on volunteers, like me, to occasionally care about the overall health of the project enough to look at the reliability reports, triage the flakes, and ping the platform teams. IMO it's fine for tier 1 platforms whose user base make them worthy of this volunteer effort, but not fine for tier 2 platforms which <1% of our users rely on to consume the scarce volunteer energy, and the platform teams should be more proactive in watching the CI results to prevent these platforms from contributing to the instablility. Otherwise this could just continue distablizing for a year or more until any volunteer cares enough to find out who's flaking the CI, all because stakeholders of a niche platform that <1% of our users use want to stick around in node-test-pull-request and assume that they would get free notifications whenever it breaks.

[mcollina]

lgtm

[legendecas]

The reliability reports are generated by dates and can not be filtered to list what's failing on a specific platform. Would it be helpful to generate a reliability summary filtered by platforms? It can help us to find out the reliability of each platform.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.19%. Comparing base (f4fcf0e) to head (822f74c).
Report is 31 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #56583      +/-   ##
==========================================
- Coverage   89.19%   89.19%   -0.01%     
==========================================
  Files         662      662              
  Lines      191762   191765       +3     
  Branches    36907    36910       +3     
==========================================
- Hits       171041   171040       -1     
+ Misses      13572    13571       -1     
- Partials     7149     7154       +5     

see 28 files with indirect coverage changes

[joyeecheung]

Would it be helpful to generate a reliability summary filtered by platforms? It can help us to find out the reliability of each platform.

SGTM though that would be a feature request in https://github.com/nodejs/node-core-utils - though I think if you just want to have an overview you could simply check out https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-master/

[legendecas]

I think if you just want to have an overview you could simply check out https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-master/

This does not provide a history of reliability over the time of each platform, does it? A reliability history by platform can provide supporting figures for us to determine if a platform is unstable.

[jasnell]

Is there an issue open to track this flaky test?

[anonrig]

Is it reasonable to expect @nodejs/platform-smartos team to track flaky tests for their platform?

[joyeecheung]

A reliability history by platform can provide supporting figures for us to determine if a platform is unstable.

Indeed, though still should be a feature request to node-core-utils.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/64486/

[lpinca]

cc: @bahamat

[lpinca]

Is there an issue open to track this flaky test?

There are only some closed issues

• Investigate flaky test-worker-prof #26401
• Failing test sequential/test-worker-prof on Apple Sillicon #36904
• investigate flaky test-worker-prof on Windows #37673
• c3c6c4f007

[bahamat]

Hi all,

Sorry for the delay, yesterday was a busy day for me. We'll be looking into this.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/64516/

[nodejs-github-bot]

Landed in 97caa4c