This repository helps you automate the process of deploying MATLAB® Parallel Server™ and MATLAB Job Scheduler (MJS) using your Azure® account.
Use this repository to deploy a compute cluster using compute, storage, and network resources hosted by Azure. The cloud resources are created using Azure Resource Manager (ARM) templates. For information about the architecture of this solution, see Learn About Cluster Architecture.
Before starting, you need the following:
- A MATLAB Parallel Server license. You can use either:
- A MATLAB Parallel Server license configured to use online licensing for MATLAB. For information on how to configure your license for cloud use, see Configure MATLAB Parallel Server Licensing for Cloud Platforms.
- A network license manager for MATLAB hosting sufficient MATLAB Parallel Server licenses for your cluster. MathWorks® provides a reference architecture to deploy a suitable Network License Manager for MATLAB on Azure or you can use an existing license manager.
- MATLAB and Parallel Computing Toolbox™ on your desktop.
- An Azure account. To configure your account to enable autoscaling (since R2022b), see Configure Azure Account.
You are responsible for the cost of the Azure services you use when you create cloud resources using this repository. Resource settings, such as Virtual Machine (VM) type, affects the cost of deployment. For cost estimates, see the pricing pages for each Azure service you use. Prices are subject to change.
To view instructions for deploying the MATLAB Parallel Server reference architecture, select a MATLAB release:
| Linux | Windows |
|---|---|
| R2024a | R2024a |
| R2023b | R2023b |
| R2023a | R2023a |
| R2022b | R2022b |
| R2022a | R2022a |
| R2021b | |
| R2021a | |
| R2020b | |
| R2020a | |
| R2019b | |
| R2019a_and_older |
This diagram illustrates the cluster architecture created by the template. When you use the Azure Resource Manager templates in this repository, it creates the MATLAB Job Scheduler and the following resources. For more information about each resource, see the Azure template reference.
Figure 2: Cluster Architecture
The following resources are created.
- Virtual Network (Microsoft.Network/virtualNetworks) The Virtual Network includes the following components:
- Subnet (Microsoft.Network/virtualNetworks/subnets)
- Network Security Group (Microsoft.Network/networkSecurityGroups)
- Each VM deployed to the Virtual Network creates the following:
- Network interface (Microsoft.Network/networkInterfaces)
- Public IP Address (Microsoft.Network/publicIPAddresses)
- Headnode VM (Microsoft.Compute/virtualMachines): A Compute VM for the cluster headnode. The MATLAB install is part of the VM image, and the job database is stored either locally on the root volume, or optionally, on a separate data disk. The headnode communicates with the clients using a secure SSL connection.
- Database Volume (optional): An optional separate data disk to store the MJS job database.
- Worker Scaling Set (Microsoft.Compute/virtualMachineScaleSets): A scale set for worker VMs to be deployed into. Clients and workers communicate using a secure SSL connection.
- Storage Account (Microsoft.Storage/storageAccounts): A standard locally redundant storage (LRS) Storage Account which hosts the File Share used to distribute files amongst cluster VMs.
- File Share created inside Storage Account: A file share inside the storage account.
- Contains the Shared Secret created by headnode to all worker VMs. The worker VMs require the Shared Secret to register and establish a secure connection with the job scheduler.
- Contains the Cluster Profile. The client machine requires the Cluster Profile to authenticate and establish a connection with the headnode.
To enable autoscaling for your cluster, you must have the following permissions:
Microsoft.Authorization/roleDefinitions/writeMicrosoft.Authorization/roleAssignments/write
To check if you have these permissions, see Check access for a user to Azure resources.
If you do not have these permissions, the Administrator or Owners of the Subscription can either:
- Assign you the built-in Azure role User Access Administrator in addition to your existing role. see Assign Azure roles.
- Create a custom role containing these permissions and attach it along with your existing role. see Create Custom roles.
Parallel Computing Toolbox and MATLAB Parallel Server let you solve computationally and data-intensive programs using MATLAB and Simulink on computer clusters, clouds, and grids. Parallel processing constructs such as parallel-for loops and code blocks, distributed arrays, parallel numerical algorithms, and message-passing functions let you implement task-parallel and data-parallel algorithms at a high level in MATLAB. To learn more, see the documentation: Parallel Computing Toolbox and MATLAB Parallel Server.
MATLAB Job Scheduler is a scheduler that ships with MATLAB Parallel Server. The scheduler coordinates the execution of jobs and distributes the tasks for evaluation to the server’s individual MATLAB sessions called workers. For more details, see How Parallel Computing Toolbox Runs a Job. The MATLAB Job Scheduler and the resources required by it are created using Azure Resource Manager templates.
Microsoft Azure is a set of cloud services which allow you to build, deploy, and manage applications hosted in Microsoft’s global network of data centers. For more information about the range of services offered by Microsoft Azure, see Azure Services. Services deployed in Azure can be created, managed, and deleted using the Azure Portal UI. For more information about the Azure Portal, see Azure Portal.
No programming or cloud experience required.
If you already have an Azure account set up and ready to use, you can start a MATLAB Parallel Server Reference Architecture cluster in less than 15 minutes. Startup time varies depending on the size of your cluster.
To learn about setting quotas, see Azure subscription and service limits.
Spot Virtual Machines allow you to use Azure VMs at a reduced cost. Azure uses Spot Virtual Machines to sell unused instances within their data centers. However, Azure can reclaim these Virtual Machines at any time. For more details, see the Azure documentation on Spot Virtual Machines.
Before enabling Spot Virtual Machines, consider these aspects:
-
Pricing: Spot Virtual Machines offer discounts compared to On-Demand Virtual Machines. The actual discount depends on the available unused capacity of the Virtual Machines within the Availability Zone (AZ). For more details, refer to the Azure documentation.
-
Behavior of your cluster when Azure reclaims a Spot Virtual Machine: Spot Virtual Machines are used only for the worker nodes, whereas the head node always uses an On-Demand Virtual Machine. This is to ensure that you do not lose any user job and task information when an Virtual Machine is reclaimed by Azure. If a spot Virtual Machine for a worker is interrupted when it is running a task, the task is marked as failed. Jobs in the queue are run when a new worker instance is available. For more information, refer to How Parallel Computing Toolbox Runs a Job.
-
The following VM sizes are not supported for Azure Spot Virtual Machines:
- B-series
- Promo versions of any size (like Dv2, NV, NC, H promo sizes)
For more information about the limitations of using Spot Virtual Machines, refer to Azure Spot Virtual Machines for Virtual Machine Scale Sets.
If you require assistance or have a request for additional features or capabilities, contact MathWorks Technical Support.
Copyright 2018 - 2024 The MathWorks, Inc.