@dblock -- taking another look at the Dockerfile, I believe it is generated from this file in the opensearch-build project.

https://github.com/opensearch-project/opensearch-build/blob/d99811b3cfab45e950cdd2ed7f004074baf7d93e/docker/release/dockerfiles/opensearch.al2.dockerfile#L110:L111

Do you think that project would be the better place to move this issue?

Hi @shikharbhardwaj we would love to work with you to improve our docker image.
I will move thie issue to build repo.

Thanks.

Hi @shikharbhardwaj, if you dont mind, we would like to invite you to open a PR if you have any solutions.
We would love to assist in the process.

Thanks!

@peterzhuamazon sounds good, I will raise a PR for this.

Hello @peterzhuamazon, raised a PR for fixing this issue here: #3320
Let me know if there is something I can help with to get this reviewed/merged
Thanks!

Added review, sorry for the delay on this @shikharbhardwaj .

Thanks.

@peterzhuamazon commenting so you can assign me

@peterzhuamazon

I was able to gracefully terminate using Tini as described above only with the path being '/bin/tini". Using the relative path "./tini" gives the error:

teddyt@Teddys-MacBook-Pro docker % docker run -it -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" chawinphat/opensearch:1.3.9 docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "./tini": stat ./tini: no such file or directory: unknown.

Thanks @chawinphat and @peterzhuamazon !
@peterzhuamazon , if you could have another review on #4217 when you get any chance, thanks!!

Hi @shikharbhardwaj we have solved this by using the exec command in the docker entrypoint file.

https://github.com/opensearch-project/opensearch-build/blob/d99811b3cfab45e950cdd2ed7f004074baf7d93e/docker/release/config/opensearch/opensearch-docker-entrypoint.sh

If the command in line 101 is started using the exec command the opensearch process will run with PID 1 and the TERM signal will be sent to this process.

Hi @shikharbhardwaj we have solved this by using the exec command in the docker entrypoint file.

https://github.com/opensearch-project/opensearch-build/blob/d99811b3cfab45e950cdd2ed7f004074baf7d93e/docker/release/config/opensearch/opensearch-docker-entrypoint.sh

If the command in line 101 is started using the exec command the opensearch process will run with PID 1 and the TERM signal will be sent to this process.

Hi @239487230597507 would you willing to share your changes?
If so, would you mind contributing the changes as a PR?

This issue has been opened for a while and we would like to resolve it soon. Thanks 😄

Any updates here? It happened already for the 2nd time that our opensearch got corrupted because the container got killed by Kubernetes (probably duing some write-operation) after the termination grace period was exceeded.

Since the author of the PR hasn't commented in almost over a year, should we open a new PR based on his branch or do you want to go with the approach by @239487230597507 ?

@cinimins @peterzhuamazon I will try to create a PR. Unfortunately I am a complete Github noob so it might take a couple of days. :D

@cinimins @peterzhuamazon I will try to create a PR. Unfortunately I am a complete Github noob so it might take a couple of days. :D

Let me know if you need any help with that! :)

To anyone who is finding this thread like myself, for the current workaround, I am using the --init option on Docker 1.13+ which uses tini and is part of https://github.com/krallin/tini.

Example from my compose file:

  opensearch:
    image: opensearchproject/opensearch:latest
    init: true