HTCondor tutorial

[carlkross]

I am wanting to work though the HTCondor totorial. There was a failure on my first attempt. I'm sorry I did not note it down.

Now I want to make a second attempt. I removed all that I could find regarding my first attempt but now I get several 409 errors saying that various features already exist.

I can provide more information if that is helpful but I wondered if there is a basic clean-up step that I am missing?

Thanks, Carl Ross

[tpdownes]

Hi Carl!

Thanks for trying the HTCondor tutorial. My guess, based on what you are telling me is that you tried the HTCondor tutorial using Cloud Shell (the command line tool in our web interface) and that you manually removed resources after provisioning a first go? Then you tried to provision a 2nd set?

1. Was the initial failure when you ran terraform or after SSH-ing to a machine?
2. Could you print the errors and the commands that produce the errors?
3. The "terraform way" of removing resources is to run terraform destroy when you are done with your work. Did you do that?
4. One way to avoid naming collisions in the toolkit is to use a different deployment_name.

You can do (3) by changing the deployment_name inside the blueprint YAML or by supplying an override at the command line.

e.g. ghpc create --vars deployment_name=new-example --vars project_id=my-project file.yaml

We'd also love to know more about what your overall objective is.

[carlkross]

Thank you for your quick and detailed reply. You ask about my objective so perhaps I will start there.

I am a (retired) physicist working as a consultant for a company that requires Monte Carlo calculations (radiation transport). Such calculations are easily run in parallel, I have them running on a desktop (8 cores) and recently on a Google Cloud VM (112 cores).

I would like to consider the option of running a cluster with, maybe, 1000 cores, for at least one problem.

I know about containers and have had my application running in one. It then seemed like it should be easy to set up a cluster. However, after looking at Google Batch and then Kubernetes I came to realize that it is not so simple.

HTCondor seems hopeful and has the advantage that I can get it running on my desktop. However, working through the tutorial was quite daunting.

I was trying to stick within my single Google Cloud project, for which I already have quota allocated but maybe that was a mistake.

Since writing, I created a second project and was able to get through to the end of the tutorial, although the example still does not run. It just seems to stay in "idle" indefinitely.

Now, to your questions:

Yes, the failures were with terraform.

I should have saved the errors, but didn't. There were several 409 errors.

I didn't know about terraform destroy. That seems useful.

I did try a different deployment name but it seemed like "htcondor-001" was hardcoded in elsewhere. But maybe that is because I didn't know about terraform destroy.

I feel I have made progress by creating a second project but I need to get the simple example to run.

Then, I need to learn how to select and expand my compute nodes. I expect that is done in the .yaml file but I still have a lot to learn.

Thanks,

Carl

[carlkross]

As a follow-up, to get the errors I saw, I restarted the tutorial.

I did "terraform destroy" but it said there was nothing left to destroy.

Here is a small file with all the 409 errors. They all seem related to things already existing (which maybe isn't really an error ...).

Tutorial_errors.txt

[tpdownes]

You might find designing a blueprint to be a useful link to look at. The idea is that blueprint_name describes a repeating pattern, while deployment_name is a specific instance of that pattern (e.g. "htcondor-pool" and "carl-test"). Using these examples, "carl-test" is a "deployment" of "htcondor-pool".

What I'm trying to show you above is that you can set the deployment_name in the YAML file or at the command-line. e.g.

ghpc create --vars deployment_name=carl-pool --vars project_id=my-project file.yaml

We have a couple paths forward:

1. Cleanup (destroy) the components that remain from your first attempt and try again "fresh"
2. Import the components that remain from your first attempt into a new deployment with the same name ("htcondor-001")

(1) is probably the simplest. (2) might teach you a bit about how terraform works if you're eager to learn.

There is also a 3rd path. I do think that we should discuss Batch would be more appropriate for you. An advantage of Batch is there may be no infrastructure for you to maintain. Are you already running your job containerized in HTCondor? Our Batch solution can create a "login node" in which you can test your job and then submit them at larger scale.

The magic of both Batch and our HTCondor solutions is that the compute nodes expand and contract based upon job submissions. So you don't need to say in advance how many nodes you want. They will be turned on to run a job and powered down when no jobs remain.

Let me know about (1) or (2); we should definitely get you through a clean terraform apply / terraform destroy cycle to ensure you don't have lingering costs.

[carlkross]

Yes, I would like to accomplish (1). As I mentioned, I tried to run the tutorial in my existing project, and I would prefer not to delete the whole project because I have a VM instance there that I use. (With hindsight I should have just created a separate project).

When I first started reading about Batch I thought it was exactly what I needed. I could see how it could run several identical jobs (or maybe the correct terminology is tasks) on a single VM instance (say, wtih 112 cores) but I did not discover how it expands to more instances.

If it can, then I should go back and understand how to do that. There seemed to be considerable less overhead than with HTCondor.

So I have a containerized version of my job that runs on my desktop. I had done that in preparation for getting some cluster configuration working (first I loooked at Kubernetes, then Batch and finally HTCondor). But no, I have not run my container in HTCondor.

A general question regarding quota and instances. So far, I haven't requested more than 112 cores so that I can run on a single c2d machine. However, if I get to the point of wanting, say, 1000 or more, what is the logic guiding a machine type? Presumably I want c2d machines and the simplest would be to select 112 cores. But the same machine is available with fewer cores. Or are you just assigned a range of whatever c2d machines are available?

Thanks,

Carl

[tpdownes]

I think the following manual steps are shortest path to cleaning up. Browse to each URL and take the action noted. You will probably need to make sure that your project is selected the first time, but then it should "stick".

• https://console.cloud.google.com/security/secret-manager
  • delete htcondor-001-pool-password
• https://console.cloud.google.com/compute/disks
  • delete access-point-boot-disk-0
  • delete central-manager-boot-disk-0
• https://console.cloud.google.com/iam-admin/serviceaccounts
  • delete htcondor-001-access
  • delete htcondor-001-cm
  • delete htcondor-001-execute
• https://console.cloud.google.com/networking/addresses/list
  • delete htcondor-pool-nat-ips-us-central1-0
  • delete htcondor-pool-nat-ips-us-central1-1
• https://console.cloud.google.com/net-services/nat/list
  • delete anything matching htcondor
• https://console.cloud.google.com/hybrid/routers/list
  • delete anything matching htcondor
• https://console.cloud.google.com/networking/networks/list
  • delete htcondor-pool

Regarding Batch and VM type selection...

Are the tasks fully independent? e.g. if you run on a 112 core machine you are running 112/N jobs, where N is the number of threads per job. You are running the containerized version under Docker?

[carlkross]

Thanks for your help!!

Everything started off fine, even my project was selected.

However, when I got to addresses/list, I first had to delete the routers and then I could delete the addresses.

Then, when I get to networks/list I see (on two single lines)

htcondor-pool 1 8896 Custom
None
2 On
us-central1 htcondor-pool-usc1
10.0.0.0/24
None
None 10.0.0.1 Off

When I attmept to delete I get:

The deletion of the subnet failed. Error: The subnetwork resource 'projects/flkforflt/regions/us-central1/subnetworks/htcondor-pool-usc1' is already being used by 'projects/flkforflt/regions/us-central1/instanceGroups/htcondor-001-x-mig'

I went through the list again but I didn't see this condor reference. Does it refer to the compute pool that was created?

Sorry I got myself in such a mess.

Carl

[carlkross]

OK - looking in Instance Groups I found htcondor-001-x-mig.

I deleted it and then could delete the networks.

Maybe now it's all gone?

[tpdownes]

I'd like to begin by suggesting you run a non-containerized job using Batch:

• https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/main/docs/cloud-batch.md
• https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/main/examples/README.md#cloud-batchyaml--
• https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/main/community/examples/cloud-batch.yaml

The blueprint above provisions:

• a shared filesystem
• a VM that acts like the Cloud Batch environment and mounts the shared filesystem at /sw
• produces a job template that reads in a file from the shared filesystem

You could remove the shared filesystem if it's not useful for you, but it might be a useful place to install non-containerized code from the VM environment. Also a useful place for output data. We can then discuss a containerized approach if you'd like.

[carlkross]

You asked about my tasks. Taking the example of my desktop (8 cores) my submissions look like:

my_exec some switches input_file_001.imp &
my_exec some switches input_file_002.imp &
on up to 008.

The input files all describe the same problem; they differ only in the random number seed so that all eight runs are statistically independent. (I have the same running on a VM instance of c2d with 112 cores).

These all run independently and complete in more or less the same run time.

Each run outputs a series of files that are combined statistically.

I will look again at Batch. Clearly, I need some sort of file system to hold the input files and the output files.

Are you suggesting that I can somehow just use my existing c2d image and run it in Batch mode? I had assumed I needed my executable in a container.

Am I right that Batch is limited to 1000 cores? This is likely fine of my application but I guess HTCondor is unlimited?

Thanks, Carl

[nick-stroud]

Hey @carlkross,

file system:
The batch example referenced by @tpdownes contains a filestore file system that will be mounted to each Batch job. In the example it is mounted under /sw but the specific local mount is configurable. The login node created by that blueprint will also have access to the filestore. You can use the login node to stage data to the shared filestore that is needed by the batch jobs and inspect outputs written to the shared filestore.

Image:
If you have a VM image (as opposed to a docker image) then you can simply reference it in these lines and your batch job will use that image. Your blueprint would include something like:

  - id: batch-job
    source: community/modules/scheduler/cloud-batch-job
    use: [network1, appfs]
    settings:
      runnable: <<Execute your job>>
      machine_type: c2d-standard-4  # updated to c2d machine type
      image:
        family: <<Your VM Image>>
        project: <<Your Project ID>>

This might not work if your job is reliant on running inside a container, or if your image is not a VM image but instead a Docker image.

Cores:
Batch restrictions are documented here. It looks like Batch is restricted to 1000 parallel tasks. Each task could be running on a machine with multiple cores, so I believe the actual core limitation would be much higher.

[carlkross]

Thanks for your help, Nick. I have my code running on a c2d machine so I should be able to use that VM image.

The terminology of jobs, task and runnables is a bit confusing to me. Before I start, can you help me with the entry

runnable: <>

I attach here an example script (with generic names) of what I run on my 8 core desktop. I can do the same on a c2d-112 machine where now I have 112 lines.

In my case, is this script my runnable?

Suppose it is and now I have 225 entries in the script. Assume I have been allocated at least 225 cores. Does Batch automatically arrange extra cores so all my jobs run in parallel?

(The limit seems clear: 1000 parallel tasks. I don't understand a task group. Anyway, we can leave that for another time).

example.txt

In real life, this is a .sh file but it said I was not allowed to paste that file type.

[nick-stroud]

I believe you have two options:

Option 1: Single line runnable: one task per core:
In this option we put the actual job execution (/path-to/my_exec) as the runnable.

General steps:

1. Deploy the blueprint
2. SSH into the login node and stage your data to /share. Note: you may have to wait a few minutes after deployment before /share is available on the login node.
3. Submit the Batch job

Things to note:

• This is using the $BATCH_TASK_INDEX to reference the file name. These are indexed from 0.
• This assumes my_exec is built into the image. Alternatively you could put in /share along with the data.
• The output data needs to be written/copied to /share or it will be lost when the job finishes.
• This works well until we want to use more than 1000 cores and we run into the task limit.
• You asked about 225 cores but multiples of the cpu count would work best so I used 224.

  modules:
  - id: network1
    source: modules/network/pre-existing-vpc

  - id: sharedfs
    source: modules/file-system/filestore
    use: [network1]
    settings: {local_mount: /share}

  - id: batch-job
    source: community/modules/scheduler/cloud-batch-job
    use: [network1, sharedfs]
    settings:
      runnable: "/path-to/my_exec -M 5 /share/input_file_`printf %04d $BATCH_TASK_INDEX`.inp > /dev/null"
      machine_type: c2d-standard-112
      task_count: 224 
      task_count_per_node: 112
      image:
        family: <<Your VM Image>>
        project: <<Your Project ID>>

  - id: batch-login
    source: community/modules/scheduler/cloud-batch-login-node
    use: [batch-job]
    outputs: [instructions]

Option 2: Runnable calls a script: one task per machine:
In this option the runnable calls your script, which would then call my_exec on 112 input files.

Things to note:

• In this option you have the additional burden of writing execution_script_0.sh and execution_script_1.sh which would each call my_exec on a different set of input files.
• The task count is an order of magnitude smaller since you only have 1 task per machine instead of 1 task per core. This would allow you to scale to much larger jobs.

  modules:
  - id: network1
    source: modules/network/pre-existing-vpc

  - id: sharedfs
    source: modules/file-system/filestore
    use: [network1]
    settings: {local_mount: /share}

  - id: batch-job
    source: community/modules/scheduler/cloud-batch-job
    use: [network1, sharedfs]
    settings:
      runnable: "/share/execution_script_${BATCH_TASK_INDEX}.sh"  # similar to your example
      machine_type: c2d-standard-112
      task_count: 2
      task_count_per_node: 1
      image:
        family: <<Your VM Image>>
        project: <<Your Project ID>>

  - id: batch-login
    source: community/modules/scheduler/cloud-batch-login-node
    use: [batch-job]
    outputs: [instructions]

In both cases you should be able to test the runnable on the login node (you will have to set the BATCH_TASK_INDEX yourself). Note that I have not actually deployed and tested these blueprints so report back if you hit issues.

[tpdownes]

@carlkross we are going to close this discussion, but please do not hesitate to respond if you would like further assistance, esp. with Batch solution. My recommendation would be to try the example without (for now) considering your application and submit a few "hello, world!" jobs.

[nick-stroud]

Hey Carl, I just want to acknowledge that I have seen your replies. I personally don't have a lot of experience with containers so I am working on getting support from others that do.

[nick-stroud]

Hey Carl,

Apologies for the slow response. I was able to reproduce your problem and have a workaround/solution.

I was able to add a startup script to the execute points which perform docker authentication with Google Container Registry and then pull a private container. This container is then available locally on the machine when the job runs. Since this script will run every time a node is booted up, it should be able to always pull the latest container.

I have not yet figured out how to apply the permissions to the condor user, so for now only containers specified in the startup script will be available (the startup script runs as root). I think it should be possible to also give the condor user authorization, although that will take some more work/testing.

The Solution

I am using the standalone helper. The other methods may also work but this is what I got working.

The module below would replace the htcondor_configure_execute_point module defined on L71 of htcondor-pool.yaml. The docker_auth_and_pull.sh is the added script.

  - id: htcondor_configure_execute_point
    source: modules/scripts/startup-script
    settings:
      runners:
      - $(htcondor_install.install_htcondor_runner)
      - $(htcondor_configure.execute_point_runner)
      - type: shell
        destination: docker_auth_and_pull.sh
        content: |
          VERSION=2.1.5
          OS=linux
          ARCH=amd64
          curl -fsSL "https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v${VERSION}/docker-credential-gcr_${OS}_${ARCH}-${VERSION}.tar.gz" \
          | tar xz docker-credential-gcr \
          && chmod +x docker-credential-gcr && sudo mv docker-credential-gcr /usr/bin/
          docker-credential-gcr configure-docker
          gcloud auth application-default login --quiet

          # List the private docker images you want to use
          #   Note you must update the project_id
          docker pull gcr.io/<project_id>/busybox 

After adding this I am able to run jobs with: docker_image = gcr.io/<project_id>/busybox.

One more note: at one point during my debugging I believe I also had to add gcloud auth application-default login --quiet to the script, but in the final form it ran without this line, so I think it was unneeded and has been excluded from the snippet above.

[carlkross]

Thanks for your efforts, Nick. It looks like this should work and I will try it with my container.

I guess docker has been installed and configured in the the Centos 7 OS? One might guess that that configuration needs to be tweaked.

My progress continues -slowly. I wanted to set up a remote desktop to the access point to have easier access to tools. I have done this with other VMs using Chrome remote desktop. However, it turns out Centos doesn't support Chrome remote desktop. I then had to learn how to add firewall rules (I couldn't even ping to the access point). Anyway, I finally have access using Remmina on my Ubuntu desktop but it still needs some fine tuning.

Happy Holidays! Carl

[cboneti]

Hi Carl,

Thanks. It sounds like you got it all working.

Providing a remote desktop module is in our radar, thanks for the feedback here.

[carlkross]

Status report:

I have spent some time working with Nick's fudge regarding the Docker container registry. At first, it seemed to work fine. That was with a single compute node.

I then moved to testing with up to five c2d-standard-8 machines. Now, some compute nodes load the container and others give me the usual permissions error.

If I put my container on the Docker public hub all five machines work fine.

It is tedious to do testing and I'm not sure what to change or to look for.

BTW, I have left out this line in the script

gcloud auth application-default login --quiet

which Nick said was not required.

Ideally, I would like to keep my container private but at the moment that doesn't seem possible.

Carl