autotest

Contains code for running end-to-end distributed testing

Test environment setup

It is highly recommended to run ATI with Python3.5+ (previous versions untested, setup guide here for 3.6 and here 3.7).

In addition, it is best practice to use virtualenvwrapper (see Basic Installation and Shell Startup File on the official docs) in order to keep different Python development environments isolated. After installing virtualenvwrapper, set up your ATI environment with mkvirtualenv -p $(which python3) autotest. To exit python virtualenv, run deactivate. To re-enter, run workon autotest (command line prompt now looks like (autotest) $ ).

If everything is correctly set up, running python -V should output Python 3.x.y where x >= 5. Install required python packages by running pip install -r requirements.txt (in the root autotest directory).

How to run

Deploy distributed cluster:

i. goto autotest/deploy/distributed and upload binary

$ ./upload_s3.sh -l <path_to_build_binary>

ii. launch cluster

$ ./deploy_cluster.sh <cluster_name> -n <num_nodes> -e t2.micro

Deploy local cluster:

i. goto autotest/deploy/local and launch cluster

$ ./launch.sh

Run tests:

i. go to autotest/autotest (if local cluster, put number of nodes as cluster_name)

(autotest) $ python run_test.py <cluster_name>

Skipping tests

If you want to skip certain tests temporarily, e.g. a test named test_to_skip, simply locate where it is defined, then add a skip decorator like so

    @skip
    def test_to_skip:

Adding Tests

All tests to be placed in dir:

autotest/autotest/test_cases

Name test case in following format:

test_<test_name>.py

Start your file with the following:

from utils import *

class TestCaseMixin:

	def test_<test_name>:
	

Each test case must return True or False to indicate if test succeeds.

Rerun after each test

If you want a test to be rerun after each test case finishes, e.g. a database sanity checker, then add a rerun_needed decorator like so

    @rerun_needed
    def test_to_rerun:

Then this test won't be run standalone, but rather after each of the other standalone tests completes. As a concrete example, if out of tests A, B, C test A has @rerun_needed, then a possible execution sequence will be (B -> A) -> (C -> A).

Example

from utils import *
from orchestration import *

class TestCaseMixin:
    
    def test_example(self):
#===========================================================================================
# 1 Action tested:
#    RPC Call (microblock_test()) to trigger action on node 

        del_id = 0        
        res = self.nodes[del_id].microblock_test()
#===========================================================================================
# 2 Feedback
#    RemoteLogsHandler grep_lines() to find pattern in node's logs
        primary_str = self.log_handler.grep_lines('ConsensusManager<MicroBlock>::OnSendRequest')[0]

#===========================================================================================
# 3 Logic
#    Parse lines and test conditions, return True/False
        entries = primary_str.split('\n')[-1]
        
        return entries == self.num_delegates