Release note generator: JavaScript heap out of memory

[anton-epishev]

We are using Classic Release pipeline and this pipeline is old, having thousands of commits.
Before adding new step to generate release notes for this pipeline I tested on my sample project to make it work. But when I tried to run it in real release the task failed.
I also tried to debug it through console tester and this failed as well (the error was different but at the same step).

Here is the piece of logs from real release:
2022-01-25T22:33:14.7371383Z The default artifact for the release was not linked to an Azure DevOps Git Repo Pull Request
2022-01-25T22:34:03.8611544Z Could not get details of any PR an error was seen: TypeError: Cannot read property 'length' of null
2022-01-25T22:34:03.8612073Z Enriching known Pull Requests
2022-01-25T22:34:03.8612912Z Total Builds: [1]
2022-01-25T22:34:03.8613286Z Total Commits: [4518]
2022-01-25T22:34:03.8613629Z Total Workitems: [1704]
2022-01-25T22:34:03.8614006Z Total Related Workitems (Parent/Children): [0]
2022-01-25T22:34:03.8614362Z Total Release Tests: [0]
2022-01-25T22:34:03.8614685Z Total Tests: [1096]
2022-01-25T22:34:03.8615033Z Total Manual Test Runs: [0]
2022-01-25T22:34:03.8615391Z Total Manual Test Configurations: [0]
2022-01-25T22:34:03.8615787Z Total Pull Requests: [0]
2022-01-25T22:34:03.8616136Z Total Indirect Pull Requests: [0]
2022-01-25T22:34:03.8616488Z Total Consumed Artifacts: [0]
2022-01-25T22:34:03.8616848Z Total WIQL Workitems: [0]
2022-01-25T22:34:11.4921824Z FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2022-01-25T22:34:12.8527575Z 1: 00007FF660F289AA v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+4506
2022-01-25T22:34:12.8528549Z 2: 00007FF660F034E6 node::MakeCallback+4534
2022-01-25T22:34:12.8528976Z 3: 00007FF660F03E60 node_module_register+2032
2022-01-25T22:34:12.8529359Z 4: 00007FF661222F4E v8::internal::FatalProcessOutOfMemory+846
2022-01-25T22:34:12.8529776Z 5: 00007FF661222E7F v8::internal::FatalProcessOutOfMemory+639
2022-01-25T22:34:12.8530248Z 6: 00007FF661409674 v8::internal::Heap::MaxHeapGrowingFactor+9620
2022-01-25T22:34:12.8530679Z 7: 00007FF661400656 v8::internal::ScavengeJob::operator=+24550
2022-01-25T22:34:12.8531201Z 8: 00007FF6613FECAC v8::internal::ScavengeJob::operator=+17980
2022-01-25T22:34:12.8531632Z 9: 00007FF6614079F7 v8::internal::Heap::MaxHeapGrowingFactor+2327
2022-01-25T22:34:12.8532067Z 10: 00007FF661407A76 v8::internal::Heap::MaxHeapGrowingFactor+2454
2022-01-25T22:34:12.8533590Z 11: 00007FF66153199B v8::internal::Factory::AllocateRawWithImmortalMap+59
2022-01-25T22:34:12.8534070Z 12: 00007FF66153444D v8::internal::Factory::NewRawTwoByteString+77
2022-01-25T22:34:12.8534538Z 13: 00007FF66157963B v8::internal::interpreter::BytecodeArrayIterator::done+5579
2022-01-25T22:34:12.8535049Z 14: 00007FF6613CBC99 v8::internal::interpreter::operator<<+54729
2022-01-25T22:34:12.8535486Z 15: 00007FF6613C9F32 v8::internal::interpreter::operator<<+47202
2022-01-25T22:34:12.8535967Z 16: 00007FF6613CA46D v8::internal::interpreter::operator<<+48541
2022-01-25T22:34:12.8536398Z 17: 00007FF6613C836D v8::internal::interpreter::operator<<+40093
2022-01-25T22:34:12.8536829Z 18: 00007FF6613CAE4E v8::internal::interpreter::operator<<+51070
2022-01-25T22:34:12.8537309Z 19: 00007FF6613C836D v8::internal::interpreter::operator<<+40093
2022-01-25T22:34:12.8537776Z 20: 00007FF6613CAE4E v8::internal::interpreter::operator<<+51070
2022-01-25T22:34:12.8538208Z 21: 00007FF6613C836D v8::internal::interpreter::operator<<+40093
2022-01-25T22:34:12.8539647Z 22: 00007FF6613CAE4E v8::internal::interpreter::operator<<+51070
2022-01-25T22:34:12.8540092Z 23: 00007FF6613C836D v8::internal::interpreter::operator<<+40093
2022-01-25T22:34:12.8541844Z 24: 00007FF6613C53A8 v8::internal::interpreter::operator<<+27864
2022-01-25T22:34:12.8542304Z 25: 00007FF6615F3936 v8::internal::WasmJs::Install+231750
2022-01-25T22:34:12.8542764Z 26: 00007FF6615F382B v8::internal::WasmJs::Install+231483
2022-01-25T22:34:12.8543151Z 27: 000002339A8DC821
2022-01-25T22:34:13.0541353Z ##[error]Exit code 134 returned from process: file name 'C:\agents\2.196.2\externals\node10\bin\node.exe', arguments '"D:\a_tasks\XplatGenerateReleaseNotes_7b0a5887-75c4-4ade-a915-f92a93c934cb\3.77.5\GenerateReleaseNotes.js"'.
2022-01-25T22:34:13.0557508Z ##[section]Finishing: Generate Release Notes based on Release Comparison API

And these errors I got in debug:
Exception has occurred: TypeError: Cannot read property 'type' of undefined
at Object. (D:\AzurePipelines\Extensions\XplatGenerateReleaseNotes\v3\ReleaseNotesFunctions.js:1566:54)
at Generator.next ()
at fulfilled (D:\AzurePipelines\Extensions\XplatGenerateReleaseNotes\v3\ReleaseNotesFunctions.js:5:58)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:93:5)

Exception has occurred: RangeError: Invalid string length
at JSON.stringify ()
at dumpJsonPayload (D:\AzurePipelines\Extensions\XplatGenerateReleaseNotes\v3\ReleaseNotesFunctions.js:1711:21)
at Object. (D:\AzurePipelines\Extensions\XplatGenerateReleaseNotes\v3\ReleaseNotesFunctions.js:1660:17)
at Generator.next ()
at fulfilled (D:\AzurePipelines\Extensions\XplatGenerateReleaseNotes\v3\ReleaseNotesFunctions.js:5:58)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:93:5)

[rfennell]

Workaround
A workaround I have used in the past when there is a lot of historic data you don't really need is as follows

1. Add a new stage to your pipeline (to contain the release note task (maybe add a basic script task, if you want)
2. Run the pipeline so the stage completes successfully
3. Add the release notes task
4. Run the pipeline, it will see the past run as successful so only build release notes since that run

But to the problem
From the logs it looks as if the failure is in the dumpJsonPayload function.

Do you have any of the dump to console or file enabled?

If you do can you try with them switched off, particularly the dump to console as I have found this a problem when the payload is larger.

[anton-epishev]

The workaround helped partially, I still see it is failing time to time with the same "JavaScript heap out of memory" error.

For the local debug I tried to disable all output but it is still failing for dumpJsonPayload method. It is hard to debug locally as it takes few hours per run.

Also I noticed that I don't have any PR's details, probably it is because another error in the logs above: "Could not get details of any PR an error was seen: TypeError: Cannot read property 'length' of null". I don't have this error when it runs in my test repo.

[rfennell]

I don't know of anything I can do over memory allocation within the Node engine running the task

I still think the best option is to create a successful build, to place a marker to not go prior to, then add the release notes generation soit only picks up changes since the successful previous run.