Support Elastic Agent

[widhalmt]

Since Elastic Agent is now GA and seems to be the future of Beats, we need to make sure, we can support it.

This is quite a big step, because we could change the way we collect data completely.

An idea how to deal with the different ways beats and agent work is the following. Keep in mind that Elastic Agent as of now only starts beats and gives them appropriate configuration. But nothing stops you from running the same beat multiple times on the same host. Starting a beat with a specific configuration is quite easy. Just have a look via ps on a system where one runs.

• Use Elastic Agent everywhere where it works well. Connect it directly to Elasticsearch/Kibana and use Fleet for management. This should be configurable via Ansible. In Fleet you'll only need to activate the integrations you want to have. Right now I don't see a point in automatically configuring integrations.
• Use classic beats wherever the Agent doesn't work really well. Integration of custom logs or logs that aren't known to integrations. Maybe even logs where there are more sophisticated Logstash rules available than there are in the integration for the tool.
• In the long run we could offer a multistep setup. Like first use Ansible to scan for running applications. When one is found, we can automatically collect the logs, roll out the Logstash configuration and / or activate the integration in Fleet via API. But that's very far from the first step.

Please ignore the first PR that's referenced. It was a very early test in the deprecated role repository.

[lcndsmr]

I would like to build an Elastic-Agent Role as independent of the other roles as possible, since then the role could be used to install the agent on other Servers, that are not part of the stack, but should be monitored.

What are your thoughts on this? @afeefghannam89 @widhalmt

[afeefghannam89]

I would like to build an Elastic-Agent Role as independent of the other roles as possible, since then the role could be used to install the agent on other Servers, that are not part of the stack, but should be monitored.

What are your thoughts on this? @afeefghannam89 @widhalmt

@dnssmr Do you mean that the Elastic Agent role would be in another GitHub repository, not in this collection?

[afeefghannam89]

@dnssmr It is a good Idea :), to build the Agent role in this collection, then the user can use the filebeat or elastic agent role or both of them is some special cases.

[lcndsmr]

@dnssmr Do you mean that the Elastic Agent role would be in another GitHub repository, not in this collection?

No i think i'd build it in the collection! But it shouldn't be dependent of the other roles! You should be able to run the role alone on any server that you want to monitor.

[widhalmt]

Yes, great idea! That's actually what the elasticstack_full_stack variable is for. When true then you can rely on the other roles of the collection being used. Means, you have your variables set, can fetch your certificates from the CA etc. If it's off, you can just install the one tool and configure it how every you want/need.

[widhalmt]

I played around a lot with the agent in a personal testing setup. What I found so far:

• Elastic wants to install the agent via tarball so Fleet is able to upgrade it by itself. I'm very torn which way we should go and I'm even thinking about providing both in the long run
• Just like with Icinga satellites, you install the agent everywhere and depending on the configuration it's either a fleet server or an agent
• Every instance has a policy set which determines what integrations to run. This is usually all done via Kibana, so I wouldn't put a lot effort into managing policies

When you have your first fleet server (a.k.a. agent) installed, you connect it to your Elastic stack. Here's my shell command derived from Elastic was providing spiced up with certificates from default paths from our collection:

./elastic-agent install --url=https://elastic801:8220   --fleet-server-es=https://elastic801:9200   --fleet-server-service token=***   --fleet-server-policy=fleet-server-policy   --certificate-authorities=/opt/es-ca/ca.crt --fleet-server-cert-key-passphrase=/opt/es-ca/elastic801.passphrase  --fleet-server-es-ca=/opt/es-ca/ca.crt --fleet-server-cert=/opt/es-ca/elastic801-es.crt.pem --fleet-server-cert-key=/opt/es-ca/elastic801-es.key.pem --fleet-server-port=8220

Note that you have to provide certificates and keys as .pem files. And you have to put the key passphrase in a file and use this as an option.

And here's an agent installation in the same setup.

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.10.3-linux-x86_64.tar.gz ;
tar xzvf elastic-agent-8.10.3-linux-x86_64.tar.gz ;
cd elastic-agent-8.10.3-linux-x86_64 ;
sudo ./elastic-agent install --url=https://elastic801:8220 --enrollment-token=*** -nf

Note that I added the installation via tarball for informational purposes.

Things that really gave me a hard time:

• You can provide the CA fingerpring instead of certificates so agents will trust the certificate from the first connect. If you don't, they might not connect at all. You need the fingerprint without :. That's wrong in several examples and seems to be an old bug. Get the certificate fingerprint like so: openssl x509 -fingerprint -sha256 -in /opt/es-ca/ca.crt | grep Fingerprint | cut -d= -f2 | tr -d ':'
• Some Integrations don't use files or sockets to connect to services, they make API calls. If the agent doesn't, via some magical way, know the CA of your service, you sometimes have a text field to set ssl. options. Use them to set CAs, verification_mode etc.
• Logs of the agent are at /opt/Elastic/Agent/data/elastic-agent-*/logs/elastic-agent-* (tarball install, at least)