security_exception: unable to authenticate user [kibana_system]

[Ghostwritten]

$  uname -a
Linux dbscale-mysql 3.10.0-1160.90.1.el7.x86_64 #1 SMP Thu May 4 15:21:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

$ vim .env
$ cat .env |grep PASS
ELASTIC_PASSWORD='elastic'
LOGSTASH_INTERNAL_PASSWORD='logstash_internal'
KIBANA_SYSTEM_PASSWORD='kibana_system'


docker-compose up -d

kibana output:

[2023-05-24T04:08:44.577+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
[2023-05-24T04:08:44.587+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 25f3f33d-eb0f-4ee7-b234-b7414c930a9c
[2023-05-24T04:08:44.698+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-24T04:08:44.698+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-05-24T04:08:44.715+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-24T04:08:44.716+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-05-24T04:08:44.723+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-24T04:08:44.734+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-24T04:08:44.738+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
[2023-05-24T04:08:44.816+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-24T04:08:44.849+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-24T04:08:44.850+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2023-05-24T04:08:44.868+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2023-05-24T04:08:45.096+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2023-05-24T04:08:45.440+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2023-05-24T04:08:45.561+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
        Root causes:
                security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
[2023-05-24T04:08:46.113+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell

Is there anything else I need to modify? The kibana log here is wrong

other err：
$ docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user logstash_internal

ERROR: Failed to reset password for the [logstash_internal] user

[antoineco]

docker-compose up setup

[antoineco]

@daniejoh not sure what your thumb down is supposed to mean. The issue description clearly shows that the setup wasn't executed. docker-elk won't work without setting up users.

[daniejoh]

That was maybe a little harsh, sorry about that.

I am having the same problem as issue author. I have run docker-compose up setup.

❯ docker-compose up setup
WARN[0000] mount of type `volume` should not define `bind` option
[+] Running 4/3
 ✔ Network docker-elk_elk                Created                                                                                                                                                               0.0s
 ✔ Volume "docker-elk_elasticsearch"     Created                                                                                                                                                               0.0s
 ✔ Container docker-elk-elasticsearch-1  Created                                                                                                                                                               0.1s
 ✔ Container docker-elk-setup-1          Created                                                                                                                                                               0.1s
Attaching to docker-elk-setup-1
docker-elk-setup-1  | [+] Waiting for availability of Elasticsearch. This can take several minutes.
docker-elk-setup-1  |    ⠿ Elasticsearch is running
docker-elk-setup-1  | [+] Waiting for initialization of built-in users
docker-elk-setup-1  |    ⠿ Built-in users were initialized
docker-elk-setup-1  | [+] Role 'heartbeat_writer'
docker-elk-setup-1  |    ⠿ Creating/updating
docker-elk-setup-1 exited with code 28


❯ docker-compose up -d
[+] Running 3/3
 ✔ Container docker-elk-elasticsearch-1  Running                                                                                                                                                               0.0s
 ✔ Container docker-elk-kibana-1         Started                                                                                                                                                               0.4s
 ✔ Container docker-elk-logstash-1       Started                                                                                                                                                               0.5s


❯ docker logs -f docker-elk-kibana-1
[2023-05-26T17:15:34.703+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2023-05-26T17:15:43.367+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
[2023-05-26T17:15:43.367+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2023-05-26T17:15:43.367+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
[2023-05-26T17:15:43.367+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
[2023-05-26T17:15:43.373+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2023-05-26T17:15:43.427+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
[2023-05-26T17:15:43.452+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-05-26T17:15:43.475+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2023-05-26T17:15:43.630+00:00][INFO ][plugins-system.standard] Setting up [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2023-05-26T17:15:43.638+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
[2023-05-26T17:15:43.640+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 986c8cc6-1384-43df-a1d3-ac88b1ab1813
[2023-05-26T17:15:43.681+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-26T17:15:43.681+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-05-26T17:15:43.695+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-26T17:15:43.695+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-05-26T17:15:43.700+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-26T17:15:43.706+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-26T17:15:43.709+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
[2023-05-26T17:15:43.751+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-26T17:15:43.769+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-26T17:15:43.770+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2023-05-26T17:15:43.784+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2023-05-26T17:15:43.946+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2023-05-26T17:15:44.114+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2023-05-26T17:15:44.141+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
	Root causes:
		security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
[2023-05-26T17:15:44.940+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_arm64/headless_shell

I see now that the setup exits with error. However I cannot figure out why. Any ideas?

Again, sorry about the thumbs down!

[antoineco]

Thanks for the additional details.

I see that the setup exited with code 28, which is an error documented as OPERATION_TIMEOUTED in curl. It's almost as if Elasticsearch became unresponsive right in the middle of the setup.

What happens if you re- run the setup?

[daniejoh]

I started again from scratch, and this is the result:

❯ docker-compose up setup
WARN[0000] mount of type `volume` should not define `bind` option
[+] Running 4/3
 ✔ Network docker-elk_elk                Created                                                                                                                                                                                       0.0s
 ✔ Volume "docker-elk_elasticsearch"     Created                                                                                                                                                                                       0.0s
 ✔ Container docker-elk-elasticsearch-1  Created                                                                                                                                                                                       0.0s
 ✔ Container docker-elk-setup-1          Created                                                                                                                                                                                       0.1s
Attaching to docker-elk-setup-1
docker-elk-setup-1  | [+] Waiting for availability of Elasticsearch. This can take several minutes.
docker-elk-setup-1  |    ⠿ Elasticsearch is running
docker-elk-setup-1  | [+] Waiting for initialization of built-in users
docker-elk-setup-1  |    ⠿ Built-in users were initialized
docker-elk-setup-1  | [+] Role 'heartbeat_writer'
docker-elk-setup-1  |    ⠿ Creating/updating
docker-elk-setup-1 exited with code 28


❯ docker-compose up setup
[+] Running 2/0
 ✔ Container docker-elk-elasticsearch-1  Running                                                                                                                                                                                       0.0s
 ✔ Container docker-elk-setup-1          Created                                                                                                                                                                                       0.0s
Attaching to docker-elk-setup-1
docker-elk-setup-1  | [+] Waiting for availability of Elasticsearch. This can take several minutes.
docker-elk-setup-1  |    ⠿ Elasticsearch is running
docker-elk-setup-1  | [+] Waiting for initialization of built-in users
docker-elk-setup-1  |    ⠍ Timed out waiting for condition
docker-elk-setup-1 exited with code 1


❯ docker-compose up setup
[+] Running 2/0
 ✔ Container docker-elk-elasticsearch-1  Running                                                                                                                                                                                       0.0s
 ✔ Container docker-elk-setup-1          Created                                                                                                                                                                                       0.0s
Attaching to docker-elk-setup-1
docker-elk-setup-1  | [+] Waiting for availability of Elasticsearch. This can take several minutes.
docker-elk-setup-1  |    ⠿ Elasticsearch is running
docker-elk-setup-1  | [+] Waiting for initialization of built-in users
docker-elk-setup-1  |    ⠍ Timed out waiting for condition
docker-elk-setup-1 exited with code 1

It consistently crashes here after about 10-20 seconds.

Environment:

❯ uname -a
Darwin MacBook-Pro-2544661.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64

❯ docker -v
Docker version 23.0.5, build bc4487a

[antoineco]

Then I'll need the full output of docker-compose logs elasticsearch to help.

[daniejoh]

The log is too big to post here.. I sent it to the email listed on your github profile.

[antoineco]

Logs received 👍
The outcome is that Elasticsearch is in read-only mode due to a disk usage threshold:

{
  "@timestamp": "2023-05-30T07:15:16.913Z",
  "log.level": "WARN",
  "message": "flood stage disk watermark [95%] exceeded on [3dcHXXVrSA25e9kysjIvBQ][elasticsearch][/usr/share/elasticsearch/data] free: 1.2gb[2%], all indices on this node will be marked read-only",
  "ecs.version": "1.2.0",
  "service.name": "ES_ECS",
  "event.dataset": "elasticsearch.server",
  "process.thread.name": "elasticsearch[elasticsearch][management][T#1]",
  "log.logger": "org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor",
  "elasticsearch.cluster.uuid": "Wbmi6A-BSouuW8oG8teiBw",
  "elasticsearch.node.id": "3dcHXXVrSA25e9kysjIvBQ",
  "elasticsearch.node.name": "elasticsearch",
  "elasticsearch.cluster.name": "docker-cluster"
}

Ref.

• Failed to authenticate user [kibana_system]: at least one primary shard for the index [.security-7] is unavailable #795 (comment)
• Authentication error on main branch: at least one primary shard for the index [.security-7] is unavailable #729 (comment)
• Cannot log in to Kibana: at least one primary shard for the index [.security-7] is unavailable #829
• elasticsearch init error: at least one primary shard for the index [.security-7] is unavailable #852

Relevant doc pages:

• https://www.elastic.co/guide/en/elasticsearch/reference/8.8/modules-cluster.html#disk-based-shard-allocation
• https://www.elastic.co/guide/en/elasticsearch/reference/8.8/fix-watermark-errors.html

Closing now because:

1. The original issue was occurring because OP forgot to run docker-compose up setup, whereas what we are discussing here is an issue with Elasticsearch.
2. This issue isn't related to docker-elk (it would happen with any distribution of Elasticsearch due to insufficient disk space) and solutions were provided in similar issue reports.

[daniejoh]

That was indeed the issue for me. Thank you for the help, and fast responses 😄