Skip to content

Commit

Permalink
fix: [TKC-1714] test suite step status main (#5164)
Browse files Browse the repository at this point in the history 
* docs: add test suite step params example (#5112)

* fix: [TKC-1714] test suite results with step params (#5156)

* fix: delete remainder of tcl licensed steps

* fix: cleanups

* dep: operator dependency

* fix: make sync backwards compatible (#5161)
  • Loading branch information
@vLia
vLia authored Mar 12, 2024
1 parent 3771c53 commit 0581501
Show file tree
Hide file tree
Showing 5 changed files with 207 additions and 258 deletions.
207 changes: 207 additions & 0 deletions docs/docs/articles/creating-test-suites.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -107,3 +107,210 @@ spec:
```
Your `Test Suite` is defined and you can start running testing workflows.

## Test Suite Steps

Test Suite Steps are the individual components or actions that make up a Test Suite. They are typically a sequence of tests that are run in a specific order. There are two types of Test Suite Steps:

Tests: These are the actual tests to be run. They could be unit tests, integration tests, functional tests, etc., depending on the context.

Delays: These are time delays inserted between tests. They are used to wait for a certain period of time before proceeding to the next test. This can be useful in situations where you need to wait for some process to complete or some condition to be met before proceeding.

Similar to running a Test, running a Test Suite Step based on a test allows for specific execution request parameters to be overwritten. Step level parameters overwrite Test Suite level parameters, which in turn overwrite Test level parameters. The Step level parameters are configurable only via CRDs at the moment.

For details on which parameters are available in the CRDs, please consult the table below:

| Parameter | Test | Test Suite | Test Step |
| ---------------------------------- | ---- | ---------- | --------- |
| name | ✓ | ✓ | |
| testSuiteName | ✓ | | |
| number | ✓ | | |
| executionLabels | ✓ | ✓ | ✓ |
| namespace | ✓ | ✓ | |
| variablesFile | ✓ | | |
| isVariablesFileUploaded | ✓ | | |
| variables | ✓ | ✓ | |
| testSecretUUID | ✓ | | |
| testSuiteSecretUUID | ✓ | | |
| args | ✓ | | ✓ |
| argsMode | ✓ | | ✓ |
| command | ✓ | | ✓ |
| image | ✓ | | |
| imagePullSecrets | ✓ | | |
| sync | ✓ | ✓ | |
| httpProxy | ✓ | ✓ | ✓ |
| httpsProxy | ✓ | ✓ | ✓ |
| negativeTest | ✓ | | |
| activeDeadlineSeconds | ✓ | | |
| artifactRequest | ✓ | | |
| jobTemplate | ✓ | ✓ | ✓ |
| jobTemplateReference | ✓ | ✓ | ✓ |
| cronJobTemplate | ✓ | ✓ | ✓ |
| cronJobTemplateReference | ✓ | ✓ | ✓ |
| preRunScript | ✓ | | |
| postRunScript | ✓ | | |
| executePostRunScriptBeforeScraping | ✓ | | |
| sourceScripts | ✓ | | |
| scraperTemplate | ✓ | ✓ | ✓ |
| scraperTemplateReference | ✓ | ✓ | ✓ |
| pvcTemplate | ✓ | ✓ | ✓ |
| pvcTemplateReference | ✓ | ✓ | ✓ |
| envConfigMaps | ✓ | | |
| envSecrets | ✓ | | |
| runningContext | ✓ | ✓ | ✓ |
| slavePodRequest | ✓ | | |
| secretUUID | | ✓ | |
| labels | | ✓ | |
| timeout | | ✓ | |

Similar to Tests and Test Suites, Test Suite Steps can also have a field of type `executionRequest` like in the example below:

```yaml
apiVersion: tests.testkube.io/v3
kind: TestSuite
metadata:
name: jmeter-special-cases
namespace: testkube
labels:
core-tests: special-cases
spec:
description: "jmeter and jmeterd executor - special-cases"
steps:
- stopOnFailure: false
execute:
- test: jmeterd-executor-smoke-custom-envs-replication
executionRequest:
args: ["-d", "-s"]
...
- stopOnFailure: false
execute:
- test: jmeterd-executor-smoke-env-value-in-args
```

The `Definition` section of each Test Suite in the Testkube UI offers the opportunity to directly edit the Test Suite CRDs. Besides that, consider also using `kubectl edit testsuite/jmeter-special-cases -n testkube` on the command line.

### Usage Example

An example of use case for test suite step parameters would be running the same K6 load test with different arguments and memory and CPU requirements.

1. Create and Configure the Test

Let's say our test CRD stored in the file `k6-test.yaml` looks the following:

```yaml
apiVersion: tests.testkube.io/v3
kind: Test
metadata:
name: k6-test-parallel
labels:
core-tests: executors
namespace: testkube
spec:
type: k6/script
content:
type: git
repository:
type: git
uri: https://github.com/kubeshop/testkube.git
branch: main
path: test/k6/executor-tests/
executionRequest:
args:
- k6-smoke-test-without-envs.js
jobTemplate: "apiVersion: batch/v1\nkind: Job\nspec:\n template:\n spec:\n containers:\n - name: \"{{ .Name }}\"\n image: {{ .Image }}\n resources:\n requests:\n memory: 128Mi\n cpu: 128m\n"
activeDeadlineSeconds: 180
```

We can apply this from the command line using:

```bash
kubectl apply -f k6-test.yaml
```

2. Run the Test

To run this test, execute:

```bash
testkube run test k6-test-parallel
```

A new Testkube execution will be created. If you investigate the new job assigned to this execution, you will see the memory and cpu limit specified in the job template was set. Checking the arguments from the `executionRequest` is also possible with:

```bash
kubectl testkube get execution k6-test-parallel-1
```

3. Create and Configure the Test Suite

We are content with the test created, but we need to make sure our application works with different kinds of loads. We could create a new Test with different parameters, but that would come with the overhead of having to manage and sync two instances of the same test. Creating a test suite makes test orchestration a more robust operation.

We have the following `k6-test-suite.yaml` file:

```yaml
apiVersion: tests.testkube.io/v3
kind: TestSuite
metadata:
name: k6-parallel
namespace: testkube
spec:
description: "k6 parallel testsuite"
steps:
- stopOnFailure: false
execute:
- test: k6-test-parallel
executionRequest:
argsMode: override
args:
- -vu
- "1"
- k6-smoke-test-without-envs.js
jobTemplate: "apiVersion: batch/v1\nkind: Job\nspec:\n template:\n spec:\n containers:\n - name: \"{{ .Name }}\"\n image: {{ .Image }}\n resources:\n requests:\n memory: 64Mi\n cpu: 128m\n"
- test: k6-test-parallel
executionRequest:
argsMode: override
args:
- -vu
- "2"
- k6-smoke-test-without-envs.js
```

Note that there are two steps in there running the same test. The difference is in their `executionRequest`. The first step is setting the number of virtual users to one and updating the jobTemplate to use a different memory requirement. The second test updates the VUs to 2.

Create the test suite with the command:

```bash
kubectl apply -f k6-test-suite.yaml
```

4. Run the Test Suite

Run the test suite with:

```bash
kubectl testkube run testsuite k6-parallel
```

The output of both of the test runs can be examined with:

```bash
testkube get execution k6-parallel-k6-test-parallel-2
testkube get execution k6-parallel-k6-test-parallel-3
```

The logs show the exact commands:

```bash
...
🔬 Executing in directory /data/repo:
$ k6 run test/k6/executor-tests/k6-smoke-test-without-envs.js -vu 1
...
🔬 Executing in directory /data/repo:
$ k6 run test/k6/executor-tests/k6-smoke-test-without-envs.js -vu 2
...
```

The job template configuration will be visible on the job level, running `kubectl get jobs -n testkube` and `kubectl get job ${job_id} -o yaml -n testkube` should be enough to check the settings.

Now we know how to increase the flexibility, reusability and scalability of your tests using test suites. By setting parameters on test suite step levels, we are making our testing automation more robust and easier to manage.
80 changes: 0 additions & 80 deletions docs/docs/testkube-pro/articles/running-parallel-tests-with-test-suite.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,83 +27,3 @@ For this test suite, we have added 5 tests that all run in parallel:
Here is an example of a Test Suite sequence with 2 tests running in parallel and, when they complete, a single test runs, then 2 addtional parallel tests:

![Test and Order of Execution](../../img/test-and-order-of-execution.png)

## Test Suite Steps

Test Suite Steps can be of two types:

1. Tests: tests to be run.
2. Delays: time delays to wait in between tests.

Similarly to running a Test, running a Test Suite Step based on a test allows for specific execution request parameters to be overwritten. Step level parameters overwrite Test Suite level parameters, which in turn overwrite Test level parameters. The Step level parameters are configurable only via CRDs at the moment.

For details on which parameters are available in the CRDs, please consult the table below:

| Parameter | Test | Test Suite | Test Step |
| ---------------------------------- | ---- | ---------- | --------- |
| name | ✓ | ✓ | |
| testSuiteName | ✓ | | |
| number | ✓ | | |
| executionLabels | ✓ | ✓ | ✓ |
| namespace | ✓ | ✓ | |
| variablesFile | ✓ | | |
| isVariablesFileUploaded | ✓ | | |
| variables | ✓ | ✓ | |
| testSecretUUID | ✓ | | |
| testSuiteSecretUUID | ✓ | | |
| args | ✓ | | ✓ |
| argsMode | ✓ | | ✓ |
| command | ✓ | | ✓ |
| image | ✓ | | |
| imagePullSecrets | ✓ | | |
| sync | ✓ | ✓ | ✓ |
| httpProxy | ✓ | ✓ | ✓ |
| httpsProxy | ✓ | ✓ | ✓ |
| negativeTest | ✓ | | |
| activeDeadlineSeconds | ✓ | | |
| artifactRequest | ✓ | | |
| jobTemplate | ✓ | ✓ | ✓ |
| jobTemplateReference | ✓ | ✓ | ✓ |
| cronJobTemplate | ✓ | ✓ | ✓ |
| cronJobTemplateReference | ✓ | ✓ | ✓ |
| preRunScript | ✓ | | |
| postRunScript | ✓ | | |
| executePostRunScriptBeforeScraping | ✓ | | |
| sourceScripts | ✓ | | |
| scraperTemplate | ✓ | ✓ | ✓ |
| scraperTemplateReference | ✓ | ✓ | ✓ |
| pvcTemplate | ✓ | ✓ | ✓ |
| pvcTemplateReference | ✓ | ✓ | ✓ |
| envConfigMaps | ✓ | | |
| envSecrets | ✓ | | |
| runningContext | ✓ | ✓ | ✓ |
| slavePodRequest | ✓ | | |
| secretUUID | | ✓ | |
| labels | | ✓ | |
| timeout | | ✓ | |

Similarly to Tests and Test Suites, Test Suite Steps can also have a field of type `executionRequest` like in the example below:

```bash
apiVersion: tests.testkube.io/v3
kind: TestSuite
metadata:
name: jmeter-special-cases
namespace: testkube
labels:
core-tests: special-cases
spec:
description: "jmeter and jmeterd executor - special-cases"
steps:
- stopOnFailure: false
execute:
- test: jmeterd-executor-smoke-custom-envs-replication
executionRequest:
args: ["-d", "-s"] // <- new field
...
- stopOnFailure: false
execute:
- test: jmeterd-executor-smoke-env-value-in-args
```

The `Definition` section of each Test Suite in the Testkube UI offers the opportunity to directly edit the Test Suite CRDs. Besides that, consider also using `kubectl edit testsuite/jmeter-special-cases -n testkube`.
68 changes: 0 additions & 68 deletions internal/app/api/v1/testsuites.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -872,71 +872,3 @@ func getExecutionsFilterFromRequest(c *fiber.Ctx) testresult.Filter {

return filter
}

// MergeStepRequest inherits step request fields with execution request
func MergeStepRequest(stepRequest *testkube.TestSuiteStepExecutionRequest, executionRequest testkube.ExecutionRequest) testkube.ExecutionRequest {
if stepRequest == nil {
return executionRequest
}
if stepRequest.ExecutionLabels != nil {
executionRequest.ExecutionLabels = stepRequest.ExecutionLabels
}

if stepRequest.Variables != nil {
executionRequest.Variables = mergeVariables(executionRequest.Variables, stepRequest.Variables)
}

if len(stepRequest.Args) != 0 {
if stepRequest.ArgsMode == string(testkube.ArgsModeTypeAppend) || stepRequest.ArgsMode == "" {
executionRequest.Args = append(executionRequest.Args, stepRequest.Args...)
}

if stepRequest.ArgsMode == string(testkube.ArgsModeTypeOverride) || stepRequest.ArgsMode == string(testkube.ArgsModeTypeReplace) {
executionRequest.Args = stepRequest.Args
}
}

if stepRequest.Command != nil {
executionRequest.Command = stepRequest.Command
}
executionRequest.Sync = stepRequest.Sync
executionRequest.HttpProxy = setStringField(executionRequest.HttpProxy, stepRequest.HttpProxy)
executionRequest.HttpsProxy = setStringField(executionRequest.HttpsProxy, stepRequest.HttpsProxy)
executionRequest.CronJobTemplate = setStringField(executionRequest.CronJobTemplate, stepRequest.CronJobTemplate)
executionRequest.CronJobTemplateReference = setStringField(executionRequest.CronJobTemplateReference, stepRequest.CronJobTemplateReference)
executionRequest.JobTemplate = setStringField(executionRequest.JobTemplate, stepRequest.JobTemplate)
executionRequest.JobTemplateReference = setStringField(executionRequest.JobTemplateReference, stepRequest.JobTemplateReference)
executionRequest.ScraperTemplate = setStringField(executionRequest.ScraperTemplate, stepRequest.ScraperTemplate)
executionRequest.ScraperTemplateReference = setStringField(executionRequest.ScraperTemplateReference, stepRequest.ScraperTemplateReference)
executionRequest.PvcTemplate = setStringField(executionRequest.PvcTemplate, stepRequest.PvcTemplate)
executionRequest.PvcTemplateReference = setStringField(executionRequest.PvcTemplate, stepRequest.PvcTemplateReference)

if stepRequest.RunningContext != nil {
executionRequest.RunningContext = &testkube.RunningContext{
Type_: string(stepRequest.RunningContext.Type_),
Context: stepRequest.RunningContext.Context,
}
}

return executionRequest
}

func setStringField(oldValue string, newValue string) string {
if newValue != "" {
return newValue
}
return oldValue
}

func mergeVariables(vars1 map[string]testkube.Variable, vars2 map[string]testkube.Variable) map[string]testkube.Variable {
variables := map[string]testkube.Variable{}
for k, v := range vars1 {
variables[k] = v
}

for k, v := range vars2 {
variables[k] = v
}

return variables
}
1 change: 0 additions & 1 deletion pkg/scheduler/testsuite_scheduler.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -650,7 +650,6 @@ func MergeStepRequest(stepRequest *testkube.TestSuiteStepExecutionRequest, execu
if stepRequest.Command != nil {
executionRequest.Command = stepRequest.Command
}
executionRequest.Sync = stepRequest.Sync
executionRequest.HttpProxy = setStringField(executionRequest.HttpProxy, stepRequest.HttpProxy)
executionRequest.HttpsProxy = setStringField(executionRequest.HttpsProxy, stepRequest.HttpsProxy)
executionRequest.CronJobTemplate = setStringField(executionRequest.CronJobTemplate, stepRequest.CronJobTemplate)
Expand Down
Loading

0 comments on commit 0581501

Please sign in to comment.