Docklet is a cloud operating system for mini-datacener. Its goal is to help multi-user share cluster resources effectively. In Docklet, every user has their own private virtual cluster (vcluster), which consists of a number of virtual Linux container nodes distributed over the physical cluster. Each vcluster is separated from others and can be operated like a real physical cluster. Therefore, most applications, especially those requiring a cluster environment, can run in vcluster seamlessly.
Users manage and use their vcluster all through web. The only client tool needed is a modern web browser supporting HTML5, like Safari, Firefox, or Chrome. The integrated jupyter notebook provides a web Workspace. In the Workspace, users can code, debug, test, and runn their programs, even visualize the outputs online. Therefore, it is ideal for data analysis and processing.
Docklet creates virtual nodes from a base image. Admins can pre-install development tools and frameworks according to their interests. The users are also free to install their specific software in their vcluster.
Docklet only need one public IP address. The vclusters are configured to use private IP address range, e.g., 172.16.0.0/16, 192.168.0.0/16, 10.0.0.0/8. A proxy is setup to help users visit their vclusters behind the firewall/gateway.
The Docklet system runtime consists of four components:
- distributed file system server
- etcd server
- docklet master
- docklet worker
Currently the Docklet system is recommend to run in Unbuntu 15.10+.
Ensure that python3.5 is the default python3 version.
Clone Docklet from github
git clone https://github.com/unias/docklet.git
Run prepare.sh from console to install depended packages and generate necessary configurations.
A root users will be created for managing the Docklet system. The
password is recorded in FS_PREFIX/local/generated_password.txt .
The main configuration file of docklet is conf/docklet.conf. Most default setting works for a single host environment.
First copy docklet.conf.template to get docklet.conf.
Pay attention to the following settings:
- NETWORK_DEVICE : the network interface to use.
- ETCD : the etcd server address. For distributed muli hosts environment, it should be one of the ETCD public server address. For single host environment, the default value should be OK.
- STORAGE : using disk or file to storage persistent data, for single host, file is convenient.
- FS_PREFIX: the working dir of docklet runtime. default is /opt/docklet.
- CLUSTER_NET: the vcluster network ip address range, default is 172.16.0.1/16. This network range should all be allocated to and managed by docklet.
- PROXY_PORT : the public port of docklet. Users use this port to visit the docklet system.
- PORTAL_URL : the portal of the system. Users access the system by visiting this address. If the system is behind a firewall, then a reverse proxy should be setup.
For multi hosts distributed environment, a distributed file system is needed to store global data. Currently, glusterfs has been tested. Lets presume the file system server export filesystem as nfs fileserver:/pub :
In each physical host to run docklet, mount fileserver:/pub to FS_PEFIX/global .
For single host environment, nothing to do.
For single host environment, start tools/etcd-one-node.sh . Some recent
Ubuntu releases have included etcd in the repository, just apt-get install etcd, and it need not to start etcd manually. For others, you
should install etcd manually.
For multi hosts distributed environment, must start dep/etcd-multi-nodes.sh in each etcd server hosts. This scripts requires users providing the etcd server address as parameters.
First, select a server with 2 network interface card, one having a public IP address/url, e.g., docklet.info; the other having a private IP address, e.g., 172.16.0.1. This server will be the master.
If it is the first time you start docklet, run bin/docklet-master init
to init and start docklet master. Otherwise, run bin/docklet-master start,
which will start master in recovery mode in background using
conf/docklet.conf.
You can check the daemon status by running bin/docklet-master status
The master logs are in FS_PREFIX/local/log/docklet-master.log and docklet-web.log.
Worker needs a basefs image to create containers.
You can create such an image with lxc-create -n test -t download,
then copy the rootfs to FS_PREFIX/local, and renamed rootfs
to basefs.
Note the jupyerhub package must be installed for this image. And the
start script tools/start_jupyter.sh should be placed at
basefs/home/jupyter.
You can check and run tools/update-basefs.sh to update basefs.
Run bin/docklet-worker start, will start worker in background.
You can check the daemon status by running bin/docklet-worker status
The log is in FS_PREFIX/local/log/docklet-worker.log
Currently, the worker must be run after the master has been started.
Open a browser, visiting the address specified by PORTAL_URL ,
e.g., http://docklet.info/
That is it.
Contributions are welcome. Please check devguide