Integration tests are failing on CentOS-7/OracleLinux-7

[dliappis]

While working on the Logstash JDK matrix job migration from Jenkins to Buildkite we noticed that the integration tests (part 1 and 2, including the PQ variant) are failing on CentOS-7 / OracleLinux 7 both on Jenkins and Buildkite.

This is the output from the Jenkins Job:

> Task :assembleTarDistribution
Error Errno::ENOENT, retrying 1/10
No such file or directory - /var/lib/jenkins/workspace/elastic+logstash+main+multijob-matrix-jdk-unix-integration-pq-1/LS_RUNTIME_JAVA/adoptiumjdk17/os/centos-7&&immutable/vendor/bundle/jruby/3.1.0/gems/psych-5.1.1.1-java/deps.lst
Installation successful

> Task :assembleTarDistribution FAILED
:assembleTarDistribution (Thread[Execution worker,5,main]) completed. Took 1 mins 4.044 secs.

FAILURE: Build failed with an exception.


Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
* Where:
Script '/var/lib/jenkins/workspace/elastic+logstash+main+multijob-matrix-jdk-unix-integration-pq-1/LS_RUNTIME_JAVA/adoptiumjdk17/os/centos-7&&immutable/rubyUtils.gradle' line: 160


You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

* What went wrong:
Execution failed for task ':assembleTarDistribution'.
See https://docs.gradle.org/7.5/userguide/command_line_interface.html#sec:command_line_warnings
> (null) undefined method `runtime' for JRuby:Module
49 actionable tasks: 47 executed, 2 up-to-date

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --debug option to get more log output.
> Run with --scan to get full insights.

We can see a similar example on Buildkite.

Steps to reproduce

Using the internal troubleshooting guide launch a CentOS-7 VM using:

./launch_linux_vm.sh --image-family platform-ingest-logstash-multi-jdk-centos-7

and then ssh into it using the emitted command. Then run (first we switch to the buildkite-agent user):

sudo -iu buildkite-agent
git clone https://github.com/elastic/logstash
cd logstash/

# unset generic JAVA_HOME
unset JAVA_HOME

# LS env vars for JDK matrix tests
export BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_17
export RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_17
export LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_17
export PATH="/opt/buildkite-agent/.rbenv/bin:$PATH"
eval "$(rbenv init -)"

ci/integration_tests.sh split 0

which after some time will fail with

> Task :assembleTarDistribution FAILED
:assembleTarDistribution (Thread[included builds,5,main]) completed. Took 41.252 secs.

FAILURE: Build failed with an exception.

* Where:
Script '/opt/buildkite-agent/logstash/rubyUtils.gradle' line: 160

* What went wrong:
Execution failed for task ':assembleTarDistribution'.
> (null) undefined method `runtime' for JRuby:Module

The above reproduction is using the latest adoptiumjdk_17 (available under ~/.java in the image), but it also fails with other JDKs e.g. with openjdk_11.
Finally, the same failure happens with the 8.11 (see Buildkite 8.11 job) where failures have been reproduced with all JDKs).

Branch 7.17 also fails (see this exhaustive Buildkite 7.17 job against all JDKs in the matrix):

• only for IT and IT PQ part 1
• in a completely different way (and much later)

Example output

    Failures:
      1) Beat Input Without TLS successfully send events
         Failure/Error: @fixture = Fixture.new(__FILE__)
         RuntimeError:
           /opt/buildkite-agent/builds/bk-agent-prod-gcp-1700560942348734153/elastic/logstash-linux-jdk-matrix-pipeline/qa/integration/services/filebeat_setup.sh FAILED with exit status pid 7607 exit 127
         Shared Example Group: "send events" called from ./specs/beats_input_spec.rb:98
         # ./services/service.rb:36:in `setup'
         # ./framework/fixture.rb:90:in `block in setup_services'
         # ./framework/fixture.rb:89:in `setup_services'
         # ./framework/fixture.rb:46:in `initialize'
         # ./specs/beats_input_spec.rb:28:in `block in <main>'
         # ./rspec.rb:32:in `<main>'
      2) Beat Input With TLS Server auth successfully send events
         Failure/Error: @fixture = Fixture.new(__FILE__)
         RuntimeError:
           /opt/buildkite-agent/builds/bk-agent-prod-gcp-1700560942348734153/elastic/logstash-linux-jdk-matrix-pipeline/qa/integration/services/filebeat_setup.sh FAILED with exit status pid 7607 exit 127
         Shared Example Group: "send events" called from ./specs/beats_input_spec.rb:131
         # ./services/service.rb:36:in `setup'
         # ./framework/fixture.rb:90:in `block in setup_services'
         # ./framework/fixture.rb:89:in `setup_services'
         # ./framework/fixture.rb:46:in `initialize'
         # ./specs/beats_input_spec.rb:28:in `block in <main>'
         # ./rspec.rb:32:in `<main>'
      3) Beat Input With TLS Mutual auth successfully send events
         Failure/Error: @fixture = Fixture.new(__FILE__)
         RuntimeError:
           /opt/buildkite-agent/builds/bk-agent-prod-gcp-1700560942348734153/elastic/logstash-linux-jdk-matrix-pipeline/qa/integration/services/filebeat_setup.sh FAILED with exit status pid 7607 exit 127
         Shared Example Group: "send events" called from ./specs/beats_input_spec.rb:162
         # ./services/service.rb:36:in `setup'
         # ./framework/fixture.rb:90:in `block in setup_services'
         # ./framework/fixture.rb:89:in `setup_services'
         # ./framework/fixture.rb:46:in `initialize'
         # ./specs/beats_input_spec.rb:28:in `block in <main>'
         # ./rspec.rb:32:in `<main>'
    Finished in 14 minutes 48 seconds (files took 5.04 seconds to load)
    32 examples, 3 failures
    Failed examples:
    rspec ./specs/beats_input_spec.rb[1:1:1] # Beat Input Without TLS successfully send events
    rspec ./specs/beats_input_spec.rb[1:2:1:1] # Beat Input With TLS Server auth successfully send events
    rspec ./specs/beats_input_spec.rb[1:2:2:1] # Beat Input With TLS Mutual auth successfully send events
    Randomized with seed 40872
Gradle Test Executor 3 finished executing tests.
> Task :logstash-integration-tests:integrationTests FAILED
org.logstash.integration.RSpecTests > rspecTests FAILED
    java.lang.AssertionError: RSpec test suit saw at least one failure.
        at org.junit.Assert.fail(Assert.java:88)
        at org.logstash.integration.RSpecTests.rspecTests(RSpecTests.java:52)