Limited functionality when combining options --policy and --update

[jlestrada]

Problem

There appears to be limited functionality when running conftest test with the combined options of --policy and --update. The issue appears to be that there is an inability to specify a directory path with policy that is other than ./policy or a custom directory with git config already set within.

The error seen is the following format

> conftest test -p /my-policy-dir -u <git-URL> /path/to/json/file
Error: running test: update policies: client get: error downloading '<git-URL>': /usr/local/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git

The issue appears to be resolved when policy is set to the default value policy or if the targeted path already has git configured.

Steps to reproduce

1. Execute conftest test with combined options

> conftest test -p /my-policy-dir -u <git-URL> /path/to/json/file

Conftest Version

> conftest --version
Conftest: 0.40.0
OPA: 0.50.2

Additional Notes and Considerations

It is worth noting that my use case involves executing conftest through Atlantis using the following documentation. Basically I am forced to have to specify the path option which is maybe the problem in itself but nothing from what i saw in the conftest documentation stated incompatibility between the two options.

If there is agreement that this is a bug and an unexpected behavior, I am happy to look into the issue and attempt to resolve with an PR.

[boranx]

hey @jlestrada

Thanks for reporting the issue!
I checked the acceptance tests for --update however, there's only one: https://github.com/open-policy-agent/conftest/blob/master/acceptance.bats#L366 in which the -p option didn't get used. From ui/ux perspective, specifying -p shouldn't be necessary to run policies against files, should it? (It seems the file is stored anyway after the detection&download on the dir where conftest is executed: https://github.com/open-policy-agent/conftest/blob/master/acceptance.bats#L367 )
On the other hand, using a command like below, I'm able to download the content to a policy folder that doesn't exist previously

./conftest test -p ./my-policy --update git::github.com/open-policy-agent/library.git examples/kubernetes/service.yaml
0 tests, 0 passed, 0 warnings, 0 failures, 0 exceptions
ll my-policy
total 20
drwxr-xr-x 13 bseref staff   416 Apr  2 14:04 ./
drwxr-xr-x 41 bseref staff  1312 Apr  2 14:04 ../
drwxr-xr-x 12 bseref staff   384 Apr  2 14:04 .git/
drwxr-xr-x  3 bseref staff    96 Apr  2 14:04 .github/
-rw-r--r--  1 bseref staff 11357 Apr  2 14:04 LICENSE
-rw-r--r--  1 bseref staff   272 Apr  2 14:04 Makefile
-rw-r--r--  1 bseref staff   995 Apr  2 14:04 README.md
drwxr-xr-x  6 bseref staff   192 Apr  2 14:04 aws/
drwxr-xr-x  4 bseref staff   128 Apr  2 14:04 build/
.
.

Could there be an issue with how git-URL is defined or how it's considered by https://github.com/open-policy-agent/conftest/blob/master/downloader/downloader.go?

[jlestrada]

Circling back to quickly confirm that your example above does work so I believe you are on the right track that there is something unique to my case. Possibly the use of a private repository?

To your point, requirement of the -p option should not be required but it should be supported to work in conjunction with --update or at least specify otherwise.

I appear to now being having some issues with reproducing the error on my end. I wonder if my git host was having intermittent issues that day I filed the ticket.

To explain my use case a bit more I am using a private gitlab repository as my host. Given the correct credentials, I dont imagine that my use case is unique to what you have specified.

I will continue to try and reproduce and give a bit more information. If i am unable to after today then I think we should just mark this closed.

Thanks for the response!

[jlestrada]

Okay I am now able to confirm but I am not entirely certain this is a bug that others will run into.

The issue I believe is only triggered when using a pre-exisiting directory space and it is outside the current working directory. Please see the following output as what i believe a way to reproduce the behavior.

❯ git clone [email protected]:open-policy-agent/conftest.git
Cloning into 'conftest'...
remote: Enumerating objects: 6232, done.
remote: Counting objects: 100% (172/172), done.
remote: Compressing objects: 100% (128/128), done.
remote: Total 6232 (delta 71), reused 125 (delta 42), pack-reused 6060
Receiving objects: 100% (6232/6232), 3.89 MiB | 7.36 MiB/s, done.
Resolving deltas: 100% (3234/3234), done.
❯ cd conftest
❯ pwd
/Users/joseestrada/conftest
❯ conftest test --update git::https://github.com/open-policy-agent/library.git --policy ~/policy examples/kubernetes/service.yaml

0 tests, 0 passed, 0 warnings, 0 failures, 0 exceptions
❯ rm -rf ~/policy
❯ mkdir ~/policy
❯ conftest test --update git::https://github.com/open-policy-agent/library.git --policy ~/policy examples/kubernetes/service.yaml
Error: running test: update policies: client get: error downloading 'https://github.com/open-policy-agent/library.git': /usr/local/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git

You can see above that the execution of the test worked the first time when the folder ~/policy was not present but when I attempted the second time after manually creating the directory path it failed.

This I believe was somewhat unique to my situation when using Atlantis. I believe the working directory is under /home/atlantis while I was attempting to use a policy directory path outside of this, ie /policies/.

TBH I dont know if this is a major issue but it does seem to show some limitation, maybe especially so when using Atlantis. Let me know what you think and if you are able to reproduce with the above output.

[boranx]

yeah, I'm able to reproduce. Basically, it occurs when there is a custom policy folder and if the update source is a git
I have debugged the commands above, and saw the err is thrown by go-getter, so it's probably go-getter's behavior: https://github.com/hashicorp/go-getter/blob/v1.6.1/client.go#L323

"error downloading 'https://github.com/open-policy-agent/library.git': /opt/homebrew/bin/git exited with 128: fatal: not a git repository (or any of the parent directories): .git\n"

so conftest is inheriting this indirectly. It might be a case where we can report to upstream too, also it might be something that got fixed on recent versions so we may consider bumping up go-getter at some point

Actually, there wouldn't be a problem if it was like the following

conftest test --update git::https://github.com/open-policy-agent/library.git --policy ~/policy examples/kubernetes/service.yaml
rm -rf ~/policy
conftest test --update git::https://github.com/open-policy-agent/library.git --policy ~/policy examples/kubernetes/service.yaml

Do you need the policy folder to be created from scratch before each execution? Otherwise, I'd say it'd be more graceful to let it be handled by go-getter rather than configuring it
if you need it, you can divide the execution to 2 phase

./conftest pull --policy ./policy2 git::https://github.com/open-policy-agent/library.git
echo "deny[...]{}" >> ./policy2/a.rego # do some ops
./conftest test -p ./policy2 path-to-json-yaml-files.yaml