[BUG] wazuh-indexer-performance-analyzer does not work as expected

[Enaraque]

Description

A community user reported for version 4.8.2 that the wazuh-indexer-performance-analyzer service was not starting correctly. This was fixed by this issue: wazuh/wazuh-packages#2321

Now, with version 4.9.0, the same user is reporting the same issue, where the changes made for the previous version are not working.

Steps to reproduce

I have installed the AIO version 4.9.0 on an Ubuntu 22.04 system. Then, I started the service with systemctl start wazuh-indexer-performance-analyzer.service

Upon checking the status, we can see that the service did not start correctly:

$ systemctl status wazuh-indexer-performance-analyzer.service
× wazuh-indexer-performance-analyzer.service - wazuh-indexer Performance Analyzer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer-performance-analyzer.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2024-09-09 13:51:32 UTC; 2s ago
    Process: 333527 ExecStart=/usr/share/wazuh-indexer/bin/wazuh-indexer-performance-analyzer/performance-analyzer-agent-cli (code=exited, status=203/EXEC)
   Main PID: 333527 (code=exited, status=203/EXEC)
        CPU: 3ms

Sep 09 13:51:31 ip-172-31-46-84 systemd[1]: wazuh-indexer-performance-analyzer.service: Main process exited, code=exited, status=203/EXEC
Sep 09 13:51:31 ip-172-31-46-84 systemd[1]: wazuh-indexer-performance-analyzer.service: Failed with result 'exit-code'.
Sep 09 13:51:32 ip-172-31-46-84 systemd[1]: wazuh-indexer-performance-analyzer.service: Scheduled restart job, restart counter is at 5.
Sep 09 13:51:32 ip-172-31-46-84 systemd[1]: Stopped wazuh-indexer Performance Analyzer.
Sep 09 13:51:32 ip-172-31-46-84 systemd[1]: wazuh-indexer-performance-analyzer.service: Start request repeated too quickly.
Sep 09 13:51:32 ip-172-31-46-84 systemd[1]: wazuh-indexer-performance-analyzer.service: Failed with result 'exit-code'.
Sep 09 13:51:32 ip-172-31-46-84 systemd[1]: Failed to start wazuh-indexer Performance Analyzer.

If I try to change the content of the file to what existed in version 4.8.2 and restart the service, it still does not work correctly:

File content

[Unit]
Description=Wazuh-indexer Performance Analyzer
PartOf=wazuh-indexer.service
After=wazuh-indexer.service

[Service]
ExecStart=/usr/share/wazuh-indexer/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli
Restart=on-failure
User=wazuh-indexer
Group=wazuh-indexer
Environment=OPENSEARCH_HOME=/usr/share/wazuh-indexer
Environment=OPENSEARCH_PATH_CONF=/etc/wazuh-indexer/
WorkingDirectory=/usr/share/wazuh-indexer

[Install]
WantedBy=wazuh-indexer.service

root@ip-172-31-46-84:/home/ubuntu# systemctl daemon-reload
root@ip-172-31-46-84:/home/ubuntu# systemctl restart wazuh-indexer-performance-analyzer.service
root@ip-172-31-46-84:/home/ubuntu# systemctl status wazuh-indexer-performance-analyzer.service
× wazuh-indexer-performance-analyzer.service - Wazuh-indexer Performance Analyzer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer-performance-analyzer.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2024-09-09 15:00:24 UTC; 2s ago
    Process: 391375 ExecStart=/usr/share/wazuh-indexer/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli (code=exited, status=203/EXEC)
   Main PID: 391375 (code=exited, status=203/EXEC)
        CPU: 2ms

Sep 09 15:00:24 ip-172-31-46-84 systemd[1]: wazuh-indexer-performance-analyzer.service: Scheduled restart job, restart counter is at 5.
Sep 09 15:00:24 ip-172-31-46-84 systemd[1]: Stopped Wazuh-indexer Performance Analyzer.
Sep 09 15:00:24 ip-172-31-46-84 systemd[1]: wazuh-indexer-performance-analyzer.service: Start request repeated too quickly.
Sep 09 15:00:24 ip-172-31-46-84 systemd[1]: wazuh-indexer-performance-analyzer.service: Failed with result 'exit-code'.
Sep 09 15:00:24 ip-172-31-46-84 systemd[1]: Failed to start Wazuh-indexer Performance Analyzer.

[AlexRuiz7]

The name of the binary has diverged

[root@rhel7 vagrant]# cd /usr/share/wazuh-indexer/bin/                                
[root@rhel7 bin]# ls
indexer-security-init.sh  opensearch-cli  opensearch-env-from-file  opensearch-node                  opensearch-plugin  opensearch-upgrade
opensearch                opensearch-env  opensearch-keystore       opensearch-performance-analyzer  opensearch-shard   systemd-entrypoint
[root@rhel7 bin]# cat /lib/systemd/system/wazuh-indexer-performance-analyzer.service
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0
#
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

[Unit]
Description=wazuh-indexer Performance Analyzer

[Service]
Type=simple
ExecStart=/usr/share/wazuh-indexer/bin/wazuh-indexer-performance-analyzer/performance-analyzer-agent-cli
Restart=on-failure
User=wazuh-indexer
Group=wazuh-indexer
EnvironmentFile=-/etc/sysconfig/wazuh-indexer
WorkingDirectory=/usr/share/wazuh-indexer

[Install]
WantedBy=multi-user.target[root@rhel7 bin]# ls opensearch-performance-analyzer/
performance-analyzer-agent  performance-analyzer-agent-cli

[byalexandrepedrosa]

@Enaraque

Thank you for bring the issue from Slack to Github!!!!

I am the user with this issue.

My environment is Ubuntu 22.04.

Some additional notes about the issue:

The:

ExecStart=/usr/share/wazuh-indexer/bin/wazuh-indexer-performance-analyzer/performance-analyzer-agent-cli

Does not exist on Ubuntu 22.04, I think the correct path would be:

ExecStart=/usr/share/wazuh-indexer/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli

Because this folder not exist: /usr/share/wazuh-indexer/bin/wazuh-indexer-performance-analyzer/

The:

EnvironmentFile=-/etc/sysconfig/wazuh-indexer

Should be:

EnvironmentFile=-/etc/default/wazuh-indexer

Because this folder not exist: /etc/sysconfig/

Installed using wazuh documentation for version 4.8.0 adding the repository then:

apt-get install wazuh-indexer

Did upgrade to 4.8.2 and got problem about paths, corrected by the help of @Enaraque.

Upgrade with:

apt-get update && apt-get upgrade -y

Did an new upgrade for 4.9.0 using same command and now, after set correct paths, application does not work.

[AlexRuiz7]

I was able to make the performance analyzer service running by setting the right path to the executable, which in fact is /usr/share/wazuh-indexer/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli.

That change unveils the original OpenSearch Performance Analyzer service file, which is what we should expect in 4.9.0. Looks like the change to the binary path was mistakenly applied through a "search and replace" command or script, leading to a non-existent path.

In order to make the service file system independent, the EnvironmentFile key has been replaced with single Environment keys, to set the environment variables required by the PA. This change unveils the OpenSearch Performance Analyzer service file before 4.9.0. The use of EnvironmentFile is not recommended.

Using a Vagrant machine with a pre-installed Wazuh 4.8.0 and then upgraded to 4.9.0, these are the commands I used to make it work:

1. Edit /usr/lib/systemd/system/wazuh-indexer-performance-analyzer.service and set the correct path to the binary in ExecStart:

[Unit]
Description=OpenSearch Performance Analyzer

[Service]
Type=simple
ExecStart=/usr/share/wazuh-indexer/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli
Restart=on-failure
User=wazuh-indexer
Group=wazuh-indexer
Environment=OPENSEARCH_HOME=/usr/share/wazuh-indexer
Environment=OPENSEARCH_PATH_CONF=/etc/wazuh-indexer/
WorkingDirectory=/usr/share/wazuh-indexer

[Install]
WantedBy=multi-user.target

2. Ensure the logs folder and all its files belong to the wazuh-indexer user and group, so the Performance Analyzer write access is granted.

mkdir /usr/share/wazuh-indexer/logs
chown -R wazuh-indexer:wazuh-indexer /usr/share/wazuh-indexer/logs

3. Reload and start the service.

systemctl daemon-reload
systemctl start wazuh-indexer-performance-analyzer
systemctl status wazuh-indexer-performance-analyzer -l

After that, the service should be up and running. Although it is, there is a Java error in the logs that seems related to the Performance Analyzer itself, and not the service file. I found an issue on the GitHub repository of the PA. There is a pull request pending to be merged to solve this problem (on a future release of OpenSearch).

[root@rhel7 vagrant]# systemctl status wazuh-indexer-performance-analyzer -l
● wazuh-indexer-performance-analyzer.service - Wazuh Indexer Performance Analyzer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer-performance-analyzer.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2024-09-10 15:41:17 UTC; 6min ago
 Main PID: 6294 (java)
   CGroup: /system.slice/wazuh-indexer-performance-analyzer.service
           └─6294 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Xms4m -Xmx64m -XX:+UseSerialGC -Dlog4j.configurationFile=/etc/wazuh-indexer/opensearch-performance-analyzer/log4j2.xml -Xms64M -Xmx64M -XX:+UseSerialGC -XX:CICompilerCount=1 -XX:-TieredCompilation -XX:InitialCodeCacheSize=4096 -XX:MaxRAM=400m -Dopensearch.path.home=/usr/share/wazuh-indexer -Dopensearch.path.conf=/etc/wazuh-indexer -Dopensearch.distribution.type=rpm -cp /usr/share/wazuh-indexer/lib/*:/usr/share/wazuh-indexer/performance-analyzer-rca/lib/* org.opensearch.performanceanalyzer.PerformanceAnalyzerApp

Sep 10 15:41:17 rhel7.localdomain systemd[1]: Started Wazuh Indexer Performance Analyzer.
Sep 10 15:41:19 rhel7.localdomain performance-analyzer-agent-cli[6294]: Exception in thread "main" java.lang.IllegalArgumentException: cannot add context to list
Sep 10 15:41:19 rhel7.localdomain performance-analyzer-agent-cli[6294]: at jdk.httpserver/sun.net.httpserver.ContextList.add(ContextList.java:37)
Sep 10 15:41:19 rhel7.localdomain performance-analyzer-agent-cli[6294]: at jdk.httpserver/sun.net.httpserver.ServerImpl.createContext(ServerImpl.java:276)
Sep 10 15:41:19 rhel7.localdomain performance-analyzer-agent-cli[6294]: at jdk.httpserver/sun.net.httpserver.HttpServerImpl.createContext(HttpServerImpl.java:74)
Sep 10 15:41:19 rhel7.localdomain performance-analyzer-agent-cli[6294]: at jdk.httpserver/sun.net.httpserver.HttpServerImpl.createContext(HttpServerImpl.java:39)
Sep 10 15:41:19 rhel7.localdomain performance-analyzer-agent-cli[6294]: at org.opensearch.performanceanalyzer.PerformanceAnalyzerApp.createClientServers(PerformanceAnalyzerApp.java:354)
Sep 10 15:41:19 rhel7.localdomain performance-analyzer-agent-cli[6294]: at org.opensearch.performanceanalyzer.PerformanceAnalyzerApp.createClientServers(PerformanceAnalyzerApp.java:319)
Sep 10 15:41:19 rhel7.localdomain performance-analyzer-agent-cli[6294]: at org.opensearch.performanceanalyzer.PerformanceAnalyzerApp.main(PerformanceAnalyzerApp.java:112)