If you use this software in your work, please cite the CloudSim Express paper
Hewage, TB, Ilager, S, Rodriguez, MA, Buyya, R. CloudSim express: A novel framework for rapid low code simulation of cloud computing environments. Softw Pract Exper. 2023; 1-18. doi: 10.1002/spe.3290
CloudSim Express is a low-code framework to implement, deploy, and share CloudSim simulations. It takes a human-friendly scripting approach to boost the productivity of CloudSim simulation life-cycle.
- Low-Code Simulation Designs
- Design simulations via a human-readable script
- Simplified Extensions
- Standardization of writing custom extensions
- Improved maintainability
- Less boilerplate code
- Deploy as a tool
- Human-readable script to update simulation scenario, and one-liner execution
- Isolation of extensions for simplified implementations, and deployments
- Implement locally, and deploy remotely
- Accelerated Idea-to-implementation
- Re-use human-readable simulation designs and isolated custom extensions
- Zero-code transferring from existing simulations
The rest of this document explore CloudSim Express in-details.
- Why did we implement CloudSim Express?
- Simulation Life Cycle Comparison: CloudSim vs CloudSim Express
- Introduction to Scripted Simulations
- Writing Extensions
- Deploying CloudSim Express
- Inner Workings of CloudSim Express
- Stability
- Releasing CloudSimExpress
- Contributing
The example usages of CloudSim Express is available in CloudSim Express - User Guide.
We observed that even though the CloudSim is a powerful tool used extensively by the research community, its development life-cycle is often a chaos. This is primarily because it's a Java library, and a standard development framework is not available. Therefore, researchers tend to use their own implementation approaches, which then lead to implementations that are practically impossible to maintain. Due to that, any spin-off research work are not able to re-use already available implementations as a starting point, rather re-implementing most of the code.
On the other hand, modern simulations quite often involves computationally heavy, large scale simulations. For example, the Azure public VM traces dataset v2 is 235 GB large and contain ~2.6M VMs with 1.9B utilization readings. In such simulations, researchers offload executions into a high performance server via a cloud service. There is not a standard way available to implement CloudSim simulations in a modularized manner to address such scenarios.
Lastly, we noticed that CloudSim simulations requires quite-a-bit of expertise in implementing Java projects. This drastically increases the time that is required to implement a simulation, and even applies to the simple examples. We think this is unfair for the research community, especially for a new user.
Focusing on these drawbacks, we are implementing CloudSim Express to revamp the entire simulation life-cycle of CloudSim. CloudSim Express introduces a human-readable scripting approach to define simulation scenarios, a standard way of writing extensions, and a simplified deployment approach. The script-based approach significantly reduces code-based implementations. In the cases of simple examples, it completely eliminates the need to implement Java code.
A typical CloudSim simulation life cycle involves following steps.
- Designing cloud infrastructure (i.e., Host count, processing elements count, etc).
- Writing extensions to simulate a specific simulation scenario. In this comparison, we implement a custom workload submission component.
- Create a new Java project, and configure CloudSim as a dependency.
- Define the cloud infrastructure by writing Java code.
// Import libraries
import org.cloudbus.cloudsim.Cloudlet;
import org.cloudbus.cloudsim.CloudletSchedulerTimeShared;
import org.cloudbus.cloudsim.Datacenter;
...
// Initialize the toolkit
int num_user=1; // number of cloud users
Calendar calendar=Calendar.getInstance();
...- Implement extensions in the same project by writing java code.
...
int id = 0;
long length = 400000;
long fileSize = 300;
long outputSize = 300;
UtilizationModel utilizationModel = new UtilizationModelFull();
...- Build the project, and execute.
...
Starting CloudSimExample1...
Initialising...
Starting CloudSim version 3.0
Datacenter_0 is starting...
...
- Download the CloudSim Express tool.
- Define cloud infrastructure in a human-readable script.
...
Zone: &Zone
name: "default zone"
datacenter: *Datacenter
broker:
variant:
className: "org.cloudbus.cloudsim.DatacenterBroker"
name: "RegionalBroker"
workloadGenerator:
variant:
className: "org.crunchycookie.research.distributed.computing.cloudsim.workload.impl.DummyWorkloadGenerator"
GlobalDatacenterNetwork:
zoneCount: 1
interZoneNetworkDescriptionFilePath: "./sample-data/aws-geo-distributed-datacenter-inter-network-data.csv"
zones:
- <<: *Zone
name: "REGIONAL_ZONE_VIRGINIA"
...- Implement extensions independently, and package as jar files. Then copy the jar files into the CloudSim Express tool.
...
public class DummyWorkloadGenerator implements CloudSimWorkloadGenerator {
public static final int TASK_COUNT = 100;
private final List<Cloudlet> cloudletList;
public DummyWorkloadGenerator() {
cloudletList = new ArrayList<>();
...- Execute the simulation via a CLI command.
...
$ sh cloudsim-express.sh
2023-04-02 01:58:46,522 [main] INFO org.cloudbus.cloudsim.express.simulator.impl.DefaultCloudSimExpressSimulator - Initializing the Low-code simulator...
2023-04-02 01:58:46,526 [main] INFO org.cloudbus.cloudsim.express.simulator.impl.DefaultCloudSimExpressSimulator - Using Extension Resolver: class org.cloudbus.cloudsim.express.resolver.impl.JARExtensionsResolver
2023-04-02 01:58:46,650 [main] INFO org.cloudbus.cloudsim.express.simulator.impl.DefaultCloudSimExpressSimulator - Using Environment Resolver: class org.cloudbus.cloudsim.express.resolver.impl.YAMLEnvironmentResolver
2023-04-02 01:58:46,650 [main] INFO org.cloudbus.cloudsim.express.simulator.impl.DefaultCloudSimExpressSimulator - Using Simulation Handler: class org.cloudbus.cloudsim.express.handler.impl.cloudsim.DefaultGlobalDatacenterNetworkHandler
2023-04-02 01:58:46,651 [main] INFO org.cloudbus.cloudsim.express.simulator.impl.DefaultCloudSimExpressSimulator - Using Scenario Manager: class org.cloudbus.cloudsim.express.manager.impl.DefaultScenarioManager
Initialising...
...
At the heart of CloudSim Express lies the simulation script, which describes the model of the simulation system. In which a system model is built ground-up using re-usable components as building blocks, via a human-readable scripting language.
We opted for YAML as the human-readable scripting language, which is extensively utilized in various applications, from defining APIs to maintaining configuration files. As a result, it is a familiar sight for CloudSim users.
The CloudSim Express currently offers following re-usable components.
- ProcessingElement
- ProcessingElementProvisioner
- Host
- Datacenter
- Broker
- Zone
- GlobalDatacenterNetwork
Our goal is to cover most of the standard cloud infrastructure. However, these components may not be sufficient for a certain simulation. For which we invite the community to contribute to the project using our standard approach of implementing such components. This approach is discussed in the following sections.
- Schematics of re-usable components
- Usage of re-usable components
- How does a re-usable component convert scripted object, into the CloudSim implementation?
- Introducing a new re-usable component
We use a separate YAML script to define schematics of re-usable components, which is the simulation-elements.yaml file. Each component is defined as an object in YAML with attributes. For example, the Host object schematics is as follows.
Host:
type: object
properties:
variant:
$ref: '#/components/schemas/Extension'
id:
type: integer
ramProvisioner:
$ref: '#/components/schemas/Extension'
bwProvisioner:
$ref: '#/components/schemas/Extension'
storage:
type: integer
processingElementList:
type: array
items:
$ref: '#/components/schemas/ProcessingElement'
vmScheduler:
$ref: '#/components/schemas/Extension'Components defined with the simulation-elements.yaml is readily available to be used by the CloudSim Express tool. For example, the same Host component is now used to define a specific host instance in the system-model script file of CloudSim Express.
Host: &Host
variant:
className: "org.cloudbus.cloudsim.Host"
copies: 1
id: -1
ramProvisioner:
className: "org.cloudbus.cloudsim.provisioners.RamProvisionerSimple"
extensionProperties:
- key: "ram"
value: "515639"
bwProvisioner:
className: "org.cloudbus.cloudsim.provisioners.BwProvisionerSimple"
extensionProperties:
- key: "bw"
value: "100000000"
vmScheduler:
className: "org.cloudbus.cloudsim.VmSchedulerTimeShared"
storage: 1000000
processingElementList: *ProcessingElementListDuring the project build, the schematics in the simulation-elements.yaml file are converted to Plain-old-java-objects (PoJo). Then, a specific handler extending Base Element Handler class is written with necessary logic to covert scripted object information into a CloudSim simulation. During runtime, the corresponding handler is selected by the tool to process all scripted objects.
In this case, the developer needs to follow the approach below.
- Define the schematic in the simulation-elements.yaml file.
- Build the project to generate the corresponding PoJo (this is automatically handled via OpenAPI 3.0 libraries).
- Write a new element handler for the component by extending the Base Element Handler.
- Add the new handler to the end of the priority list in cloudsim-express-configs.properties file.
- Build the project again to obtain the updated CloudSim Express tool. Afterward, the new component can be used in the simulation system model script.
In CloudSim Express, the re-usable components are used to write the system model of the simulation. For example, a cloud availability zone can be made up with a datacenter having 10 hosts. This can be written in YAML in the following manner.
...
Host: &Host
variant:
className: "org.cloudbus.cloudsim.Host"
copies: 1
id: -1
ramProvisioner:
className: "org.cloudbus.cloudsim.provisioners.RamProvisionerSimple"
extensionProperties:
- key: "ram"
value: "515639"
bwProvisioner:
className: "org.cloudbus.cloudsim.provisioners.BwProvisionerSimple"
extensionProperties:
- key: "bw"
value: "100000000"
vmScheduler:
className: "org.cloudbus.cloudsim.VmSchedulerTimeShared"
storage: 1000000
processingElementList: *ProcessingElementList
HostList: &HostList
- <<: *Host
variant:
className: "org.cloudbus.cloudsim.Host"
copies: 10
Datacenter: &Datacenter
variant:
className: "org.cloudbus.cloudsim.Datacenter"
characteristics: *Characteristics
vmAllocationPolicy:
className: "org.cloudbus.cloudsim.VmAllocationPolicySimple"
storage: ""
schedulingInterval: 0
name: "regional-datacenter"
Zone: &Zone
name: "default zone"
datacenter: *Datacenter
broker:
variant:
className: "org.cloudbus.cloudsim.DatacenterBroker"
name: "RegionalBroker"
workloadGenerator:
variant:
className: "org.crunchycookie.research.distributed.computing.cloudsim.workload.impl.DummyWorkloadGenerator"
extensionProperties:
- key: "workloadPercentage"
value: "11"
- key: "isRandomPercentage"
value: "true"
- key: "dataFolder"
value: "../external-dependencies/data/workload/bitbrain-dataset/fastStorage"
...Notice how the system model is built from ground up, each component wrapping another.
The top-most component in this situation is the Zone. Once we have that, we use a reserved
component called SimulationSystemModel to let the CloudSim Express to know that which component defines the
simulation system model.
...
SimulationSystemModel:
name: 'Zone'
...We introduce a standard approach to write extensions via system model YAML file using the keyword, variant. It is an attribute of
a re-usable component. For example, a datacenter is an extension point. An extension of a datacenter is identified by
its class name. Therefore in the system model script, a datacenter can have the attribute variant. In the example below,
we use the default datacenter class org.cloudbus.cloudsim.Datacenter.
...
Datacenter: &Datacenter
variant:
className: "org.cloudbus.cloudsim.Datacenter"
extensionProperties:
- key: "sampleProperty"
- value: "11"
characteristics: *Characteristics
vmAllocationPolicy:
className: "org.cloudbus.cloudsim.VmAllocationPolicySimple"
storage: ""
schedulingInterval: 0
name: "regional-datacenter"
...The schematic of the variant attribute is defined with the Extension component in simulation-elements.yaml.
Extension:
type: object
properties:
className:
type: string
copies:
type: integer
extensionProperties:
type: array
items:
$ref: '#/components/schemas/Property'As observed, any number of custom properties can be supplied to an extension as key, value pairs. This allows managing custom properties too, via the system model script file.
In order to consume the defined properties, the researcher must implement the interface CloudSimExpressExtension.java. For the previous example, if the researcher wants to use a custom datacenter class, then that class must implement the above-mentioned interface, which provides APIs to consume the properties.
The extensions are packed as jars and copied into the extensions folder of the CloudSim Express tool. During the
runtime, they are loaded into the JVM and looked-up using the class name defined in the script.
Once this project is built, the CloudSim Express tool is released into the /release-artifacts folder as a zipped file.
Once unzipped, the tool has the following file structure.
├── cloudsim-express.sh
├── configs.properties
├── extensions
│ └── workload-generators-0.1.jar
├── log4j2.properties
├── logs
│ └── cloudsim-1680488172390.log
├── sample-data
│ └── aws-geo-distributed-datacenter-inter-network-data.csv
├── scenarios
│ └── system-model.yaml
└── simulator.jar
extensionsfolder- Copy all custom class file jars into this folder.
scenariosfolder- The
system-model.yamlfile is used to define the simulation system model in YAML using the re-usable components.
- The
cloudsim-express.shfile- The tool is executed alongside with this file using the command,
sh ./cloudsim-express.sh
- The tool is executed alongside with this file using the command,
This root folder can be copied into any remote machine and simply executed in the same manner. Most of the simulation parameter updates can be done via the system model script file, and re-run the simulation without even compiling java code. Also, introduction of an extension can also be easily done by copying the extension jar and configuring the system model script file.
- Environment Resolver: Process the system model script file into PoJo objects, and assign a handler to proceed.
- Extensions Resolver: Read and create instances from extension classes during runtime.
- Element Handler: Convert each component from script to corresponding java implementations.
- Scenario Manager: Find and executes the element handler corresponds to the root simulation scenario.
- Simulation manager: Initialize CloudSim and starts executing the scenario.
Most of the components from above are extensible. To use a customized version, implement the
corresponding class, pack as a jar, and copy into the extensions folder. Then change the
default component to the extended version by modifying the configs.properties file.
The current version of CloudSim Express is fairly stable, and we are proud to say that it has already contributed to a potential research work. However, it is not extensively tested regarding error handling. A misconfiguration could well run into a weird error.
We would appreciate opening GitHub issues for such cases. If you are able to, please provide as much information as you can, to help us to tackle the issue.
You can start a remote debugging session with CloudSim Express by executing the simulation with
java -jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:6006 simulator.jar configs.properties
command. Connect via the port 6006. This will allow you to collect
additional information, and even to fix the issue! In that case, please
follow the Contributing guidelines and send us a pull request.
Once built, the /release-artifacts folder contains the artifacts.
- Open a GitHub issue and clearly describe the issue/feature-request.
- Fork the project, implement and test, and create a pull request.
- Wait until automated build passes with your PR.
- Upon successfully PR with the automatic build, a CloudsLab organization member will merge your contribution.
Please kindly refer to the project licence.