Container registries

[carlkross]

As indicated in a previous discussion chain, I want to run an HTCondor cluster using Docker containers.

I have success when I put my container on the public Docker Hub.

However, I have not had success when putting my container in a private Google container registry. I can create the registry and push my container.

However, efforts to pull the container generally lead to permission errors. The only way I have succeeded in pulling it is to use gcloud credential helper. However, this approach means going to a web site and copying a key so is not practical for use with a cluster.

I would have thought that, within a given project, if I can push a container, permissions and access rights should be automatically set so that I can pull the image.

Perhaps I am missing soemthing simple ...

Carl

[tpdownes]

@carlkross I don't believe the previous recommendation (#660 (reply in thread)) required downloading a key. It is run as root during startup of the compute nodes and should have access to the service account. Is this the method you've been trying?

[carlkross]

Thanks for your help.

So Nick had suggested a fudge that required running a startup script on each execute node.

At first, I thought this was working, but on subsequent tests I was getting permission errors on accessing the container registry.

I have tried various modifications to the script but at this point I am giving up on using the gcr for container storage. It is time-consuming to destroy and rebuild the cluster and I don't fully understand all (maybe any??) of the permission issues I encounter.

Yes, I know I can have a private Docker hub and I have one created for other purposes. However, it seemed complicated to set up access to this in Google Cloud. Maybe I should go back and review that as an option.

I am giving up on the storage bucket idea. Transfers seem relatively slow. Instead, I will create a tar file of results and just download that for local processing.

I am pleased that I now have Condor running. It is very well suited to my problem. Once I realized I could set up the cluster by downloading the toolkit to my local machine, things advanced more quickly.

However, the whole permissions/rules business in Google Cloud is very daunting for a beginner. Ideally, for a single user with a single project and a single cluster, everything should just "work". E.g., I wanted to use "gcloud compute scp ..." to transfer files to and from the access point. I had to do a lot of exploring to discover that, yes, port 22 was enabled, but only for a restricted range of IP addresses. No doubt there is a good reason for that but for a beginner it creates headaches.

Carl

[tpdownes]

The default VPC created by our modules/network/vpc module enables SSH through Identity-Aware Proxy (IAP), a service that tunnels SSH through a secure connection authenticated by your Google Cloud account. The advantage being that you can connect via IAP without putting your machine on the public internet (where it will be attacked by bad actors). The related gcloud compute ssh/scp commands accept an argument --tunnel-through-iap which will ensure that it tries that approach first.

If you follow the documentation for the vpc module, it will take you to a link that shows an example firewall rule you can add (the firewall_rules setting on the vpc module) which allows all SSH ingress. That said, we will consider adding an extra setting which optionally allows all SSH in addition to the IAP SSH so that you don't need to know the structure of the firewall rule object.

If you want to add this to an existing blueprint, set the argument of firewall_rules to something similar to the example.

We plan some investments in the HTCondor modules generally and I will be sure that those changes are documented when they are released. You can "watch" for new releases by managing your notifications for this repo in the top-upper-right of most GitHub pages (when viewing the Toolkit repo). The icon looks like an eye and you can use the drop-down menu to customize the notifications you receive.