Skip to content

Commit

Permalink
Refactor qa/acceptance tests to get away from vagrant (#15696)
Browse files Browse the repository at this point in the history 
This commit modernizes the qa/acceptance (packaging) test framework by
moving away from Vagrant and having the tests operate locally.

As we are migrating to Buildkite, the expectation is that those tests
will run on dedicated vms thus removing the necessity of vagrant.

Relates: elastic/ingest-dev#1722
  • Loading branch information
@dliappis
dliappis authored Jan 8, 2024
1 parent 73c87a4 commit cebe4a7
Show file tree
Hide file tree
Showing 47 changed files with 469 additions and 1,006 deletions.
102 changes: 35 additions & 67 deletions ci/acceptance_tests.sh
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
#!/usr/bin/env bash
set -e
set -x
set -eo pipefail

function get_package_type {
# determines OS packaging system; at the moment either rpm or deb
source /etc/os-release

if [[ $ID == "ubuntu" || $ID == "debian" || $ID_LIKE == "debian" ]]; then
PACKAGE_TYPE="deb"
elif [[ $ID_LIKE == *"rhel"* || $ID_LIKE == *"fedora"* || $ID_LIKE == *"suse"* ]]; then
PACKAGE_TYPE="rpm"
else
echo "^^^ +++ Unsupported Linux distribution [$ID]. Acceptance packaging tests only support deb or rpm based distributions. Exiting."
exit 1
fi
}

# Since we are using the system jruby, we need to make sure our jvm process
# uses at least 1g of memory, If we don't do this we can get OOM issues when
Expand All @@ -13,78 +26,33 @@ if [ -n "$BUILD_JAVA_HOME" ]; then
GRADLE_OPTS="$GRADLE_OPTS -Dorg.gradle.java.home=$BUILD_JAVA_HOME"
fi


SELECTED_TEST_SUITE=$1

# The acceptance test in our CI infrastructure doesn't clear the workspace between run
# this mean the lock of the Gemfile can be sticky from a previous run, before generating any package
# we will clear them out to make sure we use the latest version of theses files
# If we don't do this we will run into gem Conflict error.
[ -f Gemfile ] && rm Gemfile
[ -f Gemfile.lock ] && rm Gemfile.lock

# When running these tests in a Jenkins matrix, in parallel, once one Vagrant job is done, the Jenkins ProcessTreeKiller will kill any other Vagrant processes with the same
# BUILD_ID unless you set this magic flag: https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller
export BUILD_ID=dontKillMe

LS_HOME="$PWD"
QA_DIR="$PWD/qa"

# Always run the halt, even if the test times out or an exit is sent
cleanup() {

cd $QA_DIR
bundle check || bundle install
bundle exec rake qa:vm:halt
}
trap cleanup EXIT

# Cleanup any stale VMs from old jobs first
cleanup
cd $LS_HOME

if [[ $SELECTED_TEST_SUITE == $"redhat" ]]; then
echo "Generating the RPM, make sure you start with a clean environment before generating other packages."
cd $LS_HOME
./gradlew clean bootstrap
rake artifact:rpm
echo "Acceptance: Installing dependencies"
cd $QA_DIR
bundle install
get_package_type

echo "Acceptance: Running the tests"
bundle exec rake qa:vm:setup["redhat"]
bundle exec rake qa:vm:ssh_config
bundle exec rake qa:acceptance:redhat
bundle exec rake qa:vm:halt["redhat"]
elif [[ $SELECTED_TEST_SUITE == $"debian" ]]; then
echo "Generating the DEB, make sure you start with a clean environment before generating other packages."
cd $LS_HOME
# in CI (Buildkite), packaging artifacts are pre-built from a previous step
if [[ $BUILDKITE == true ]]; then
export LS_ARTIFACTS_PATH="$HOME/build"
echo "--- Downloading artifacts from \"build/*${PACKAGE_TYPE}\" to $LS_ARTIFACTS_PATH"
set -x
# also creates build/ under $HOME
buildkite-agent artifact download "build/*${PACKAGE_TYPE}" $HOME
set +x
echo "--- Running gradle"
./gradlew clean bootstrap
rake artifact:deb
echo "Acceptance: Installing dependencies"
cd $QA_DIR
bundle install

echo "Acceptance: Running the tests"
bundle exec rake qa:vm:setup["debian"]
bundle exec rake qa:vm:ssh_config
bundle exec rake qa:acceptance:debian
bundle exec rake qa:vm:halt["debian"]
elif [[ $SELECTED_TEST_SUITE == $"all" ]]; then
echo "Building Logstash artifacts"
cd $LS_HOME
else
echo "--- Detected a distribution that supports \033[33m[$PACKAGE_TYPE]\033[0m packages. Running gradle."
./gradlew clean bootstrap
rake artifact:all

echo "Acceptance: Installing dependencies"
cd $QA_DIR
bundle install

echo "Acceptance: Running the tests"
bundle exec rake qa:vm:setup
bundle exec rake qa:vm:ssh_config
bundle exec rake qa:acceptance:all
bundle exec rake qa:vm:halt
echo "--- Building Logstash artifacts"
rake artifact:$PACKAGE_TYPE
fi

echo "--- Acceptance: Installing dependencies"
cd $QA_DIR
bundle install

echo "--- Acceptance: Running the tests"
rake qa:acceptance:all
1 change: 0 additions & 1 deletion qa/Gemfile
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
source "https://rubygems.org"
gem "runner-tool", :git => "https://github.com/purbon/runner-tool.git"
gem "rspec", "~> 3.1.0"
gem "rake"
gem "stud"
Expand Down
172 changes: 38 additions & 134 deletions qa/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,130 +1,44 @@
## Acceptance test Framework

Welcome to the acceptance test framework for Logstash. In this small README we
describe its features and the steps necessary for setting up your environment.
The acceptance test framework for Logstash is intended to test the functionality of packages (`.deb`, `.rpm`)
on various supported platforms.

### Setup your environment
In this small README we describe its features and the steps necessary for executing it and adding new tests.

In summary this test framework is composed of:

* A collection of rspec helpers and matchers that make creating tests
easy.
* This rspecs helpers execute commands over SSH to a set of machines.
* The tests are run, for now, as vagrant (virtualbox provided) machines.

As of this, you need to have installed:

* The latest version vagrant (=> 1.8.1)
* Virtualbox as VM provider (=> 5.0)

Is important to notice that the first time you set everything up, or when a
new VM is added, there is the need to download the box (this will
take a while depending on your internet speed).


### Running Tests

It is possible to run the full suite of the acceptance test with the codebase by
running the command `ci/acceptance_tests.sh`, this command will generate the artifacts, bootstrap
the VM and run the tests.
### Description

This test are based on a collection of Vagrant defined VM's where the
different test are going to be executed, so first setup necessary is to
have vagrant properly available, see https://www.vagrantup.com/ for
details on how to install it.

_Inside the `qa` directory_

First of all execute the command `bundle` this will pull the necessary
dependencies in your environment, after this is done, this is the collection of task available for you:

```
skywalker% rake -T
rake qa:acceptance:all # Run all acceptance
rake qa:acceptance:debian # Run acceptance test in debian machines
rake qa:acceptance:redhat # Run acceptance test in redhat machines
rake qa:acceptance:single[machine] # Run one single machine acceptance test
rake qa:acceptance:suse # Run acceptance test in suse machines
rake qa:vm:halt[platform] # Halt all VM's involved in the acceptance test round
rake qa:vm:setup[platform] # Bootstrap all the VM's used for this tests
rake qa:vm:ssh_config # Generate a valid ssh-config
```

Important to be aware that using any of this commands:

```
rake qa:acceptance:all # Run all acceptance
rake qa:acceptance:debian # Run acceptance test in debian machines
rake qa:acceptance:redhat # Run acceptance test in redhat machines
rake qa:acceptance:suse # Run acceptance test in suse machines
```

before you *will have to bootstrap* all selected machines, you can do
that using the `rake qa:vm:setup[platform]` task. This is done like this
as bootstrap imply setting up the VM'S and this might take some time and
you might only want to this once.

In the future we might add new rake tasks to do all at once, but for now you can use the script under
`ci/ci_acceptance.sh` to do all at once.
In summary this test framework is composed of:

For local testing purposes, is recommended to not run all together, pick your target and run with the single machine command, If you're willing to run on single one, you should use:
* A collection of rspec helpers and matchers that make creating tests easy.
* Rspecs helpers that execute commands.

```
rake qa:acceptance:single[machine] # Run one single machine acceptance test
```
The tests are expected to be executed on the target environment e.g. an Ubuntu 22.04 vm.

### How to run tests
### Running tests/Prerequisites

If you are *running this test for first time*, you will need to setup
your VM's first, you can do that using either `vagrant up` or `rake qa:vm:setup[platform]`.
To run the tests from a fresh Logstash checkout, you need to:

In this framework we're using ssh to connect to a collection of Vagrant
machines, so first and most important is to generate a valid ssh config
file, this could be done running `rake qa:vm:ssh_config`. When this task
is finished a file named `.vm_ssh_config` will be generated with all the
necessary information to connect with the different machines.
1. `./gradlew clean boostrap`
2a. Build the necessary package artifacts e.g. `rake artifact:deb`
**OR**
2b. Supply a directory where pregenerated package artifacts exit via the `LS_ARTIFACTS_PATH` environment variable (relative and absolute paths are supported).
3. `cd qa`
4. `bundle install`

Now is time to run your test and to do that we have different options:
Now you are ready to kick off the tests:

* rake qa:acceptance:all # Run all acceptance
* rake qa:acceptance:debian # Run acceptance test in debian machines
* rake qa:acceptance:redhat # Run acceptance test in redhat machines
* rake qa:acceptance:suse # Run acceptance test in suse machines
* rake qa:acceptance:single[machine] # Run one single machine acceptance test
5. `rake qa:acceptance:all`.

Generally speaking this are complex tests so they take a long time to
finish completely, if you look for faster feedback see at the end of this
README how to run fewer tests.
Steps 1, 2b, 3, 4, 5 are executed by the `ci/acceptance_tests.sh` script.

## Architecture of the Framework

If you wanna know more about how this framework works, here is your
section of information.

### Directory structure

* ```acceptance/``` here it goes all the specs definitions.
* ```config``` inside you can find all config files, for now only the
platform definition.
* ```rspec``` here stay all framework parts necessary to get the test
running, you will find the commands, the rspec matchers and a
collection of useful helpers for your test.
* ```sys``` a collection of bash scripts used to bootstrap the machines.
* ```vagrant``` classes and modules used to help us running vagrant.

### The platform configuration file

Located inside the config directory there is the platforms.json which is used to define the different platforms we test with.
Important bits here are:

* `latest` key defines the latest published version of LS release which is used to test the package upgrade scenario.
* inside the `platforms` key you will find the list of current available
OS we tests with, this include the box name, their type and if they
have to go under specific bootstrap scripts (see ```specific: true ```
in the platform definition).

This file is the one that you will use to know about different OS's
testes, add new ones, etc..
* ```acceptance/```: all the specs definitions.
* ```rspec```: all framework parts necessary to get the test
running. Includes the commands, the rspec matchers and a
collection of useful helpers.

### I want to add a test, what should I do?

Expand All @@ -143,45 +57,35 @@ including the different moving parts we encounter in the framework.


```
config = ServiceTester.configuration
config.servers.each do |address|
##
# ServiceTester::Artifact is the component used to interact with the
# destination machineri and the one that keep the necessary logic
# for it.
##
logstash = ServiceTester::Artifact.new(address, config.lookup[address])
logstash = ServiceTester::Artifact.new()
## your test code goes here.
end
# example:
it_behaves_like "installable_with_jdk", logstash
it_behaves_like "updated", logstash, from_release_branch="7.17"
```

this is important because as you know we test with different machines,
so the build out artifact will be the component necessary to run the
actions with the destination machine.
Inside the `rspec` directory you will find a
collection of commands, organized per operating system, which
will let you operate and get your tests done.

but this is the main parts, to run your test you need the framework
located inside the ```rspec``` directory. Here you will find a
collection of commands, properly organized per operating system, that
will let you operate and get your tests done. But don't freak out, we
got all logic necessary to select the right one for your test.

You'll probably find enough supporting classes for different platforms, but if not, feel free to add it.
You'll probably find enough supporting classes for different platforms, but if not, feel free to add more.

FYI, this is how a command looks like:
An example of an install command on debian looks like:

```
def installed?(hosts, package)
def installed?(package)
stdout = ""
at(hosts, {in: :serial}) do |host|
cmd = sudo_exec!("dpkg -s #{package}")
stdout = cmd.stdout
end
cmd = sudo_exec!("dpkg -s #{package}")
stdout = cmd.stdout
stdout.match(/^Package: #{package}$/)
stdout.match(/^Status: install ok installed$/)
end
end
```

this is how we run operations and wrap them as ruby code.

### Running a test (detailed level)
Expand Down
Loading

0 comments on commit cebe4a7

Please sign in to comment.