Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Add monitor to watch for stale connections and evict them #373

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add monitor to watch for stale connections and evict them #373

wants to merge 1 commit into from

Conversation

wkruse
Copy link

@wkruse wkruse commented Apr 30, 2021

Issue #, if available:

Description of changes:

From time to time we are losing the alerts triggered by custom webhook. It looks like connections in the pool are closed after some time. When an alert is triggered and closed/broken connection from the pool is used, we see following stacktrace in the logs:

javax.net.ssl.SSLException: Connection reset
        at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1680) ~[?:?]
        at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1054) ~[?:?]
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.10.jar:4.5.10]
        at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.getHttpResponse(DestinationHttpClient.java:135) ~[alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.execute(DestinationHttpClient.java:98) ~[alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:42) [alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:29) [alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.lambda$publish$0(Notification.java:44) [alerting-notification-1.13.1.0.jar:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.publish(Notification.java:42) [alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.model.destination.Destination.publish(Destination.kt:296) [opendistro-alerting-1.13.1.0.jar:1.13.1.0]
        at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner$runAction$2.invokeSuspend(MonitorRunner.kt:540) [opendistro-alerting-1.13.1.0.jar:1.13.1.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.3.72.jar:1.3.72-release-468 (1.3.72)]
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
        Suppressed: java.net.SocketException: Broken pipe
                at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:420) ~[?:?]
                at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:440) ~[?:?]
                at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:826) ~[?:?]
                at java.net.Socket$SocketOutputStream.write(Socket.java:1051) ~[?:?]
                at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:82) ~[?:?]
                at sun.security.ssl.TransportContext.fatal(TransportContext.java:400) ~[?:?]
                at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
                at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
                at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1680) ~[?:?]
                at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1054) ~[?:?]
                at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.10.jar:4.5.10]
                at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.getHttpResponse(DestinationHttpClient.java:135) ~[alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.execute(DestinationHttpClient.java:98) ~[alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:42) [alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:29) [alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.lambda$publish$0(Notification.java:44) [alerting-notification-1.13.1.0.jar:?]
                at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.publish(Notification.java:42) [alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.model.destination.Destination.publish(Destination.kt:296) [opendistro-alerting-1.13.1.0.jar:1.13.1.0]
                at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner$runAction$2.invokeSuspend(MonitorRunner.kt:540) [opendistro-alerting-1.13.1.0.jar:1.13.1.0]
                at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.3.72.jar:1.3.72-release-468 (1.3.72)]
                at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
                at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
                at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
                at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: java.net.SocketException: Connection reset
        at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323) ~[?:?]
        at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) ~[?:?]
        at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) ~[?:?]
        at java.net.Socket$SocketInputStream.read(Socket.java:981) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1434) ~[?:?]
        at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1038) ~[?:?]
        ... 31 more
[2021-04-16T06:05:44,506][INFO ][c.a.o.a.m.MonitorRunResult] [opendistro-0] Internal error: javax.net.ssl.SSLException: Connection reset. See the Elasticsearch.log for details
java.lang.IllegalStateException: javax.net.ssl.SSLException: Connection reset
        at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:46) ~[alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:29) ~[alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.lambda$publish$0(Notification.java:44) ~[alerting-notification-1.13.1.0.jar:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.publish(Notification.java:42) ~[alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.model.destination.Destination.publish(Destination.kt:296) ~[opendistro-alerting-1.13.1.0.jar:1.13.1.0]
        at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner$runAction$2.invokeSuspend(MonitorRunner.kt:540) ~[opendistro-alerting-1.13.1.0.jar:1.13.1.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.3.72.jar:1.3.72-release-468 (1.3.72)]
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: javax.net.ssl.SSLException: Connection reset
        at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1680) ~[?:?]
        at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1054) ~[?:?]
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.10.jar:4.5.10]
        at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.getHttpResponse(DestinationHttpClient.java:135) ~[alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.execute(DestinationHttpClient.java:98) ~[alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:42) ~[alerting-notification-1.13.1.0.jar:?]
        ... 11 more
        Suppressed: java.net.SocketException: Broken pipe
                at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:420) ~[?:?]
                at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:440) ~[?:?]
                at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:826) ~[?:?]
                at java.net.Socket$SocketOutputStream.write(Socket.java:1051) ~[?:?]
                at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:82) ~[?:?]
                at sun.security.ssl.TransportContext.fatal(TransportContext.java:400) ~[?:?]
                at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
                at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
                at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1680) ~[?:?]
                at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1054) ~[?:?]
                at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.12.jar:4.4.12]
                at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.10.jar:4.5.10]
                at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.10.jar:4.5.10]
                at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.getHttpResponse(DestinationHttpClient.java:135) ~[alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.execute(DestinationHttpClient.java:98) ~[alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:42) ~[alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:29) ~[alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.lambda$publish$0(Notification.java:44) ~[alerting-notification-1.13.1.0.jar:?]
                at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
                at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.publish(Notification.java:42) ~[alerting-notification-1.13.1.0.jar:?]
                at com.amazon.opendistroforelasticsearch.alerting.model.destination.Destination.publish(Destination.kt:296) ~[opendistro-alerting-1.13.1.0.jar:1.13.1.0]
                at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner$runAction$2.invokeSuspend(MonitorRunner.kt:540) ~[opendistro-alerting-1.13.1.0.jar:1.13.1.0]
                at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.3.72.jar:1.3.72-release-468 (1.3.72)]
                at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
                at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
                at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
                at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: java.net.SocketException: Connection reset
        at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323) ~[?:?]
        at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) ~[?:?]
        at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) ~[?:?]
        at java.net.Socket$SocketInputStream.read(Socket.java:981) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472) ~[?:?]
        at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) ~[?:?]
        at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1434) ~[?:?]
        at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1038) ~[?:?]
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.12.jar:4.4.12]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.10.jar:4.5.10]
        at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.getHttpResponse(DestinationHttpClient.java:135) ~[alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.execute(DestinationHttpClient.java:98) ~[alerting-notification-1.13.1.0.jar:?]
        at com.amazon.opendistroforelasticsearch.alerting.destination.factory.CustomWebhookDestinationFactory.publish(CustomWebhookDestinationFactory.java:42) ~[alerting-notification-1.13.1.0.jar:?]
        ... 11 more

No retry happens, the alert get lost.

The proposed solution is to monitor stale connections and and evict them.

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or

(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or

(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.

(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@wkruse