Megalinter doesn't work with Azure Devops automated PR build runs #3921
Comments
|
@karl18 wow, that's quite a setup, you are probably much more expert about Azure than I am :)
You can see the discussion on this issue #1125 When you run manually a pipeline, it is done with your user, that probably has more rights than Azure Build Services user, that could explain why it does not crash Can you try to give more rights to the Azure build service ? |
|
You might also try to make a new git checkout step just before MegaLinter, because it seems to not find a repo according to the error |
|
@nvuillam Thanks for the compliment, I've been working on this project for quite a while now! :) Your doubts about why we see the error and the troubleshooting steps are lovely and make much sense to me! But unluckily, at the end of the day,
I was asking if any of the ENV vars in Appreciate it .. we could also engage any Azure Pipelines expert that you know ! ;) |
|
I think that BUILD_SOURCEBRANCHNAME should be the source branch of the PR :) About experts... I think @EtienneDeneuve @DariuszPorowski @riosengineer @lumarel are quite experts in Azure... maybe they can help ? ^^ |
|
I only ever run MegaLinter in In ADO, this does work for me and has done across many different orgs/projects - it will only pickup the changed files which is great - plus it means it's quicker to run as the full repo scan can take sometime depending on repo/file sizes and linters enabled. I notice that the MegaLinter task is running off a custom ADO agent pool called: Has it ever worked for you? |
|
Hi @riosengineer, Basically these Agents are self-hosted ones used as a resort for the MegaLinter Image, instead of pulling the image each time a build run ! I tried to use Microsoft hosted Agents + checking the project permissions to the Agents (Build_Service +Project granted Administrator on these Agents) and still the issue is not solved! If it's related to a permission issue, the Job won't be running initially! Of course MegaLinter is working fine except when it's a PR automated (maybe I suggest checking my question one more time) e.g. when I open a pull request and the Policy queue the pipeline, here's my problem! I always see Regarding |
|
I didn't follow closely the whole thread, but doesn't it seem similar to the limitation on GitHub Actions where when an automated action is done by the GITHUB_TOKEN, it doesn't trigger any more CI (to prevent an infinite loop and probably security too). The way around it is to use a Personal Access Token, where the actions are done as if it was that user. |
You can use Like: trigger: none
pool:
vmImage: ubuntu-latest
jobs:
- job: MegaLinter
displayName: MegaLinter
steps:
- checkout: self
- script: docker pull oxsecurity/megalinter:v8
displayName: Pull MegaLinter
- script: |
docker run -v $(System.DefaultWorkingDirectory):/tmp/lint \
--env-file <(env | grep -e SYSTEM_ -e BUILD_ -e TF_ -e AGENT_) \
-e VALIDATE_ALL_CODEBASE=false \
-e DISABLE_LINTERS=REPOSITORY_CHECKOV,SPELL_CSPELL,YAML_YAMLLINT,COPYPASTE_JSCPD,ARM_ARM_TTK,REPOSITORY_KICS,REPOSITORY_DEVSKIM \
-e SYSTEM_ACCESSTOKEN=$(System.AccessToken) \
-e GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \
oxsecurity/megalinter:v8
displayName: Run MegaLinter
- task: PublishPipelineArtifact@1
condition: succeededOrFailed()
displayName: Upload MegaLinter reports
inputs:
targetPath: $(System.DefaultWorkingDirectory)/megalinter-reports/
artifactName: MegaLinterReportI actually don't need to set any special permissions for this to work. You don't or shouldn't have to amend the build service perms, at least on an MS Hosted pipeline run anyway. The above saved in a As for your actual issue, I am not sure, sorry, but hopefully the above helps. Maybe try the above yaml with a hosted agent, and temporarily set it as the build validation pipeline to test it works ok? |
|
Hi @riosengineer I tried several And somehow I had the same issue for I appreciate any other Idea which I could try out on Azure pipelines, which I don't know! Appreciate your response ! Note: I also checked #1125 , #3421 @nvuillam @echoix @riosengineer Regarding the main thread issue's, Yay! I figured out how to solve it 2 weeks ago ;) : |
Nope, the snippet I pasted above is all I have on a completely fresh ADO project with no other changes (besides making the build validation policy to trigger on PR). Maybe worth trying in a fresh project to rule out permission problems. |
|
Hi @riosengineer @nvuillam @echoix, I managed to make it working by changing the location where megalinter checks for the updated files in [/tmp/lint] using git diff. I just realized that 
I changed this line from:
Only one thing now left is |
|
@karl18 amazing finding ! :) Please can you make a PR to update the doc once you unlocked MegaLinter's greatest mystery with Azure ? ^^ It's around here :) https://github.com/oxsecurity/megalinter/blob/main/README.md#azure-pipelines ( don't bother propagating in /docs, it's done by automated build scripts) |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this issue should stay open, please remove the |
github-actions
bot
added
the
O: stale 🤖
|
Its not stale, I'm still working on the |
github-actions
bot
removed
the
O: stale 🤖
Hi @nvuillam I run megalinter in Azure Devops. Everything looks fine except 3 questions\ issues:
However, If I ran the PR pipeline manually it works fine and this very strange to me !
Maybe a good check would be to move to
LOG_LEVEL: DEBUGand see what are the difference between automated PRs run and manually triggered env vars passed to megalinter when the script runs!The only main\ obvious difference i see is (according to Diffchecker) between both env vars on both pipelines are:
on a manually triggered one I see:
not sure if these could affect the megalinter.py script to run as expected as
BUILD_SOURCEBRANCHNAMEcould be a wrong input or something similar??this is whether its set to
trueorfalsein either "docker run cmd" or in the "megalinter.yaml" config file itself. I tried with severalfetchDepth:0&1&2for both self repo but nothing helped!FILTER_REGEX_EXCLUDE: | .*fp-cicd-shared-scripts/.*I want to exclude this folder "fp-cicd-shared-scripts" from ALL linters that might be used currently or in the furture, is it necessary to define it for each and every linter the regex exclude? or just use
FILTER_REGEX_EXCLUDE:is sufficient for all linters? If yes, do i define the regex correctly?Also last thing worth mentioning, that i do some merge for the linter configuration with
ruamel.yamland at the end I get some file with the name: .merged-megalinter.yaml`which has the end results\ full merge for all linters to run along with the developers desired linters and some framework linters for the configuration file, but thats different story as it doesn't affects the pipeline, it works fine apart from the automated PR builds which the issue is about!I really appreciate your look and further investigation!
Sorry for pasting long configs & thanks in advance! :)
here's my configuration ".megalinter.yaml":
here's my pipeline configuration "pr_docker_build.yml":
this pipeline runs a template to be able to merge the linter files as mentioned earlier, here is that file ci_helm_merge_megalinter_configs.yml:
pipeline error (ONLY for PR automated builds) [check the vars i talked about earlier]):
The text was updated successfully, but these errors were encountered: