You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After a version bump, we tend to see CI failures for integration tests for a significant amount of time after, as we wait for upstream artifacts to be created for the newly bumped version of the stack. This leaves us in a state where we have unreliable CI for potentially days until all of the artifacts are created.
During this period, we may make changes that effectively go untested, and get approved for expediency, or because the changes are deemed "low risk". However, these "low risk" changes, can still introduce unexpected bugs such as #15687.
We should consider adding a set of integration tests that do not require upstream artifacts, and will run regardless of whether the upstream artifacts are available. There are a number of our integration tests that fit this bill - such as the cli smoke tests for logstash-plugin. We could also take advantage of the serverless offering to add integration tests that require an Elasticsearch instance.
The text was updated successfully, but these errors were encountered:
robbavey
changed the title
Add "smoke test" integration tests that do not require on upstream artifacts
Add "smoke test" integration tests that do not require upstream artifacts
Dec 15, 2023
After a version bump, we tend to see CI failures for integration tests for a significant amount of time after, as we wait for upstream artifacts to be created for the newly bumped version of the stack. This leaves us in a state where we have unreliable CI for potentially days until all of the artifacts are created.
During this period, we may make changes that effectively go untested, and get approved for expediency, or because the changes are deemed "low risk". However, these "low risk" changes, can still introduce unexpected bugs such as #15687.
We should consider adding a set of integration tests that do not require upstream artifacts, and will run regardless of whether the upstream artifacts are available. There are a number of our integration tests that fit this bill - such as the cli smoke tests for
logstash-plugin
. We could also take advantage of the serverless offering to add integration tests that require an Elasticsearch instance.The text was updated successfully, but these errors were encountered: