AWS cluster management

[s-canchi]

How can one request a cluster of computers and mange them for jobs similar to a HPC system on AWS ? For example HPC system uses slurm to automate and manage jobs. What is the AWS equivalent ?

[josesanchez1815]

AWS has the ability to run Kubernetes as it is an open source software that allows you to deploy and manage containerized applications at scale.

Kubernetes works by managing a cluster of compute instances and scheduling containers to run on the cluster based on available compute resources and the resource requirements of each container.

Here is a link with some more information
https://aws.amazon.com/kubernetes/

[josesanchez1815]

Another way you can request a cluster of computers on AWS is through the Amazon Elastic Container Service.

An Amazon ECS cluster is a logical grouping of tasks or services. If you are running tasks or services that use the EC2 launch type, a cluster is also a grouping of container instances. If you are using capacity providers, a cluster is also a logical grouping of capacity providers. When you first use Amazon ECS, a default cluster is created for you, but you can create multiple clusters in an account to keep your resources separate.

Here is a link with more detailed information
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html

[ctb]

Thanks jose!

While both of these are indeed answers, they are kind of terrible answers from the perspective of researchers - Kubernetes and ECS are both quite technically difficult to use and have a different target audience. I would suggest using a workflow system to describe your workflow and then delegating cluster management to the workflow, e.g. snakemake supports kubernetes.

@luizirber may have more specific responses, too.