Stratio Acceptance Test library
- Cucumber for test definition
- TestNG for execution
- AsyncHTTP Client
- Selenium
- JSONPath
- AssertJ
Testing runtime to rule over Stratio's acceptance tests
These tests are suposed to be executed as follows:
mvn verify [-D\<ENV_VAR>=\<VALUE>] [-Dit.test=\<TEST_TO_EXECUTE>|-Dgroups=\<GROUP_TO_EXECUTE>]
Examples:
single class execution
mvn verify -DSECS=AGENT_LIST=1,2 -Dit.test=com.stratio.qa.ATests.LoopTagAspectIT
group execution
mvn verify -DSECS=5 -Dgroups=hol
As part of BDT implementation, there are a couple of AspectJ aspects which may be useful for your scenarios:
- RunOnTagAspect:
@runOnEnv(METRIC_HOST)
@skipOnEnv(METRIC_HOST)
An AspectJ aspect could handle cucumber tags so that the annotated scenario will be overseen if no environment variable METRIC_HOST exists and has value (so that no traces of it execution show up)
Example @runOnEnv:
@runOnEnv(SECS)
Scenario: Dummy scenario
And I wait '${SECS}' seconds
This scenario will ONLY be executed if environment vairable SECS is defined.
Example @skipOnEnv:
@skipOnEnv(SECS_2)
Scenario: Dummy scenario
And I wait '${SECS}' seconds
This scenario will be omitted if environment vairable SECS_2 is defined.
- IgnoreTagAspect
An AspectJ aspect that allows the skipping of an scenario or a feature. To do so a tag must be used before the scenario or the feature. Additionally an ignored reason can be set.
@ignore @manual
@ignore @unimplemented
@ignore @toocomplex
@ignore @tillfixed(DCS-XXX)
This last ignored reason is associated to a ticket in Jira. After executing the test class the ticket link is shown as execution result.
- IncludeTagAspect
An AspectJ aspect that includes an scenario before the taged one. It manages parameters as well.
@include(feature:<feature>,scenario:<scenario>)
@include(feature:<feature>,scenario:<scenario>,params:<params>)
Examples:
Simple example
@include(feature:sample.feature,scenario:Not so dummy scenario)
Scenario: Dummy scenario
And I wait '${SECS}' seconds
Predefined param
@include(feature:sample.feature,scenario:Not so dummy scenario,params:[param1:1])
Scenario: Dummy scenario
And I wait '${SECS}' seconds
Environment variable(maven command)
@include(feature:sample.feature,scenario:Not so dummy scenario,params:[param1:${test}])
Scenario: Dummy scenario
And I wait '${SECS}' seconds
Saved variable param
@include(feature:sample.feature,scenario:Not so dummy scenario,params:[param1:!{test}])
Scenario: Dummy scenario
And I wait '${SECS}' seconds
Included Scenario
Scenario: Included scenario
And I wait '<param1>' seconds
- LoopTagAspect
An AspectJ aspect that allows looping over scenarios. It provides two functionalities:
@loop
Using this tag before an scenario will convert this scenario into an scenario outline, changing parameter defined "NAME" for every element in the environment variable list received.
@loop(LIST_PARAM,NAME)
Being LIST_PARAM: -DLIST_PARAM=elem1,elem2,elem3
Examples:
@loop(AGENT_LIST,VAR_NAME)
Scenario: write <VAR_NAME> a file the final result of the scenario.
Given I run 'echo <VAR_NAME> >> testOutput.txt' locally
@loop(AGENT_LIST,VAR_NAME)
Scenario: With scenarios outlines and datatables
Given I create file 'testSOATtag<VAR_NAME.id>.json' based on 'schemas/simple<VAR_NAME>.json' as 'json' with:
| $.a | REPLACE | @{JSON.schemas/empty.json} | object |
Given I save '@{JSON.testSOATtag<VAR_NAME.id>.json}' in variable 'VAR'
Then I run '[ "!{VAR}" = "{"a":{}}" ]' locally
-DAGENT_LIST=1,2
More examples can be found in Loop feature
@multiloop
Using this tag before an scenario will convert this scenario into an scenario outline, changing parameter defined names for every element in the environment variable lists received. It creates every possibility between provided lists.
@multiloop(LIST_PARAM1=>NAME1,LIST_PARAM2=>NAME2,LIST_PARAM3=>NAME3)
Being LIST_PARAMX: -DLIST_PARAMX=elem1,elem2,elem3
Examples:
@multiloop(AGENT_LIST=>VAR_NAME)
Scenario: write <VAR_NAME> a file the final result of the scenario.
Given I run 'echo <VAR_NAME> >> testOutput.txt' locally
Given -DAGENT_LIST=1,2, it will lead to an scenario outline with 2 different examples (one for each value in AGENT_LIST)
File will contain these 2 lines:
1
2
Notice that this case @multiloop(AGENT_LIST=>VAR_NAME) is equivalent to @loop(AGENT_LIST,VAR_NAME)
@multiloop(SERVER_LIST=>SERVER_NAME,CLIENT_LIST=>CLIENT_NAME)
Scenario: write <SERVER_NAME> and <CLIENT_NAME> into a file.
Given I run 'echo "<SERVER_NAME>,<CLIENT_NAME>" >> testMultiloopOutput.txt' locally
Given -DSERVER_LIST=server1,server2 -DCLIENT_LIST=client1,client2,client3, it will lead to an scenario outline with 6 different examples (one for each possible tuple of values, mixing SERVER_LIST and CLIENT_LIST)
File will contain these 6 lines:
server1,client1
server2,client1
server1,client2
server2,client2
server1,client3
server2,client3
More examples can be found in Multiloop feature
- Background Tag
An AspectJ aspect included in loopTagAspect that allows conditional backgrounds. Its used inside the Background label as can be seen in the examples:
@background(VAR) // Beginning of conditional block of steps
Given X
When Y
Then Z
@/background // End of block
Being VAR: -DVAR=value
Examples:
Background:
Given I run '[ "SHOULD_RUN" = "SHOULD_RUN" ]' locally
@background(WAIT_NO)
Given I run '[ "SHOULD_RUN" = "FAIL_RUN" ]' locally
Given I run '[ "SHOULD_RUN" = "FAIL_RUN" ]' locally
Given I run '[ "SHOULD_RUN" = "FAIL_RUN" ]' locally
@/background
Given I run '[ "SHOULD_RUN" = "SHOULD_RUN" ]' locally
If the test above its executed WITH -DWAIT_NO=value then the background will be:
Background:
Given I run '[ "SHOULD_RUN" = "SHOULD_RUN" ]' locally
Given I run '[ "SHOULD_RUN" = "FAIL_RUN" ]' locally
Given I run '[ "SHOULD_RUN" = "FAIL_RUN" ]' locally
Given I run '[ "SHOULD_RUN" = "FAIL_RUN" ]' locally
Given I run '[ "SHOULD_RUN" = "SHOULD_RUN" ]' locally
On the other hand, if it is executed WITHOUT the environment variable, the background will be:
Background:
Given I run '[ "SHOULD_RUN" = "SHOULD_RUN" ]' locally
Given I run '[ "SHOULD_RUN" = "SHOULD_RUN" ]' locally
In conclusion, if environment variable is defined the code below the tag would be included as part of the background, if not, it will be omitted.
More examples can be found in Background feature
- Important Tag
This tag pretends to be an improvement that quit execution after tagged scenariuo fails. Examples:
Feature: Example Quit after tag
Scenario: Keeps executing next scenario
Given something
When fail
@important
Scenario: important and fails
Given something
When fail
Then something
In addition, this tag can be used the other way. By defining the property -Dquietasdefault=false every scenario turns "important" so a @notimportant tag can be used to prevent fast failure like this:
Feature: Example Quit after tag
Scenario: Keeps executing next scenario
Given something
When fail
@notimportant
Scenario: important and fails
Given something
When fail
Then something
Finally, the @important tag will make the rest of the Features of the Test Group to be skipped, but this behavior
can be overwritten using the tag @continueOnPreviousFeatureFailure in the Feature(s) that you want to keep being executed.
Example:
Feature: Failing feature with important tag
@important
Scenario: important and fails
Given I fail
Feature: Skipped feature
Scenario: Skipped
Then this feature will be skipped because the previous failed
@continueOnPreviousFeatureFailure
Feature: Feature that will continue
Scenario: Will continue
Then this feature will be exeucted