Automatically validating version updates in the release process

[tenthe]

Hi all,

to follow up on the discussion in #1169, I would like to talk about how we can verify that the StreamPipes version update is working as expected. In particular, how we can automate this process for all future versions.

In [1] we already have some scripts to test more complex scenarios. They have to be triggered manually and are not included in the nightly automated tests.

Maybe we can discuss the following points:

• When do we want to perform these tests?
  • Should it always be possible to migrate from the latest release version to the latest dev version? If yes we could run the tests every night.
• How would such tests look like?
  • Maybe we can start the old version and create some resources, then update the system and check if everything still works

What are your thoughts?

Cheers,
Philipp

[1] https://github.com/apache/streampipes/tree/dev/ui/cypress/tests/experimental

[bossenti]

Should it always be possible to migrate from the latest release version to the latest dev version?

I guess this is sometimes hard to achieve, but it's definetely a great aim which I'd like to follow.
Daily tests would also provide direct feedback to applied changes and we can tackle them immediately.
If we only run these tests as part of the release process, e.g., we might need to fix a lot of issues at once and it might get very complex.

Maybe we can start the old version and create some resources, then update the system and check if everything still works

Well this sounds reasonable, could be realized as a GitHub workflow
But is it that easy as it sounds?

[tenthe]

I guess this is sometimes hard to achieve, but it's definetely a great aim which I'd like to follow. Daily tests would also provide direct feedback to applied changes and we can tackle them immediately. If we only run these tests as part of the release process, e.g., we might need to fix a lot of issues at once and it might get very complex.

I totally agree, I am not sure if it will be possible, but I think we should try to achieve this.
We can also start with a small step into this direction.

Well this sounds reasonable, could be realized as a GitHub workflow But is it that easy as it sounds?

I think that should be possible. If you look at the nightly cypress validation action [1]. There we start StreamPipes with the command docker-compose up --build -d.

Now the workflow for the new validation could look something like this:

1. Start old StreamPipes version with docker-compose up (use last Release version)
2. Trigger Cypress tests to setup some StreamPipes resources
3. Stop StreamPipes (docker-compose down)
4. Start latest StreamPipes version with docker-compose up --build -d
5. Trigger cypress tests to validate all resources are still there and functional.

I'd suggest that we do this in two steps. First, we need to implement the cypress tests. Therefore, we can use a similar setup as in [2]. Once the tests work we run them in an action.

[1] https://github.com/apache/streampipes/blob/dev/.github/workflows/cypress-test.yml
[2] https://github.com/apache/streampipes/tree/dev/ui/cypress/tests/experimental

[bossenti]

Thats a good plan! Would you be open to create an issue for the cypress tests? That would be great :)