First call performance in Windows hosted runner (Azure pipelines) #27404
Comments
jessehouwing
added
the
bug
microsoft-github-policy-service
bot
added
customer-reported
|
Thank you for opening this issue, we will look into it. |
|
Updated with logs. From what I gather looking at the debug logs, the hosted runner doesn't have a The Visual Studio warmup moves a bunch of files from the provisioning user profile to the default user profile. My proposed change in the warmup scripts does exactly that after running It also looks like extensions aren't fully compiled until they're called at least once. That'd why I'm adding Is there anything else we can do to speed up first-call performance? |
|
Confirming a massive speedup after carrying over the following folders: actions/runner-images#8294 (comment) Still seeing a perf hit on first call of |
|
Hello, runner images team here. the issue with "C:\Users\VssAdministrator.azure" is that it resides in ephemeral user profile. is it possible to store that folder somewhere ? not in user profile |
In my PR the folder is copied to the default profile and will thus be copied to the new user that actually runs the cli. That also works. Getting all of the data out of the profile would be even better. |
that would be poor man approach indeed, if it is supported I would prefer to store files in some permanent folder another question is "what are folders to be kept". they were discovered, may they change in future ? |
Looks like the I changed my PR to use that instead. Where would you put it, https://learn.microsoft.com/en-us/cli/azure/azure-cli-configuration#cli-configuration-file |
|
that looks good. give me some time to test it |
|
Found 2 more environment variables while searching the sources for the azure-devops extension. Added those and cleaned up the pull request. |
PramodValavala-MSFT
added
Service Attention
|
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @v-anvashist, @V-hmusukula. |
|
With the world moving to more and more ephemeral runners (if only for the added security benefits), first run performance of tools will become more and more important. The following things would be things to consider: Have the ability to install
Or offer a single command we can run to ensure all of the above after installation. |
Describe the bug
When calling
azand especiallyaz pipelineson the hosted runner the first call takes a performance hit varying between 60 and 90 seconds on windows and 10-15 seconds on linux.That time is spent rebuilding the command index and interpreting/compiling python code for AZ and te extensions.
I've been looking into improving performance by 'warming up' the agent during image generation.
We've measured improvements in the Windows Hosted Runner by:
~/.azureto a system shared location~/.azure-devopsto a system shared location~/.azure-devops/python/cacheto a system shared locationaz --helpduring image generation to force a command index rebuildaz --helpandaz devops --helpto force python interpretation/compilationaz devops loginto force installation of keyringCombined this results in a 1 minute improvement of the first-run on Windows. It results in a ~12s improvement on Ubuntu.
Not every tool vendor should have to figure out how to do this, plus the behavior might change from
az's end at any time. Theactions/actions-runnerteam isn't keen on taking on having to configureazas part of image generation.With the world moving to more and more ephemeral runners (if only for the added security benefits), first run performance of tools will become more and more important. The following things would be things to consider.
Have the ability to install
azin ephemeral mode or forazto detect it's running in a CICD environment which would:az --versionperforms an update check #27437azthoughaz extension addaz loginis still attempted "just in case".--detectto use environment variables available when running in Azure Pipelines azure-devops-cli-extension#1368azruns in a CI/CD environment.go,rustandkotlin.Or offer a single command we can run to ensure all of the above after installation.
Related command
az pipelines runs show --idErrors
First execution takes way too long.
Issue script & Debug output
Script being called in azure-pipelines windows hosted runner:
Debug logs
Expected behavior
Faster first run time in hosted agent by calling az in a way that preloads most extensions.
Environment Summary
azure-cli 2.52.0
core 2.52.0
telemetry 1.1.0
Extensions:
azure-devops 0.26.0
Dependencies:
msal 1.24.0b1
azure-mgmt-resource 23.1.0b2
Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'
Python (Windows) 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]
Additional context
No response
The text was updated successfully, but these errors were encountered: