Child host becomes unreachable in redundant group when parent is unreachable #10014

jjuanino · 2024-03-03T16:48:22Z

Describe the bug

Dear community,
unless I am misunderstanding something, redundant groups does not work as expected.
Consider the following setup:

   poc_grand_parent 
             |
             |
             | (classic dependency)
             |
             | 
       poc_parent_0                                     poc_parent_1
        \                                                /
          \        (redundant dependency)              /   
            \                                        / 
              \                                    /
                 \                               /
                    \                          /
                        \                 /
                             poc_child

poc_parent_0 has a classical dependency with poc_grand_parent.
poc_child has a redundant dependency with poc_parent_0 and poc_parent_1

The issue is as follows: when poc_grand_parent becomes down, poc_parent_0 becomes unreachable (as usual), but poc_child also, which is unexpected.

To Reproduce

Consider the following setup:

object Host "poc_grand_parent" { check_command = "dummy"; vars.dummy_state = 2; }
object Host "poc_parent_0" { check_command = "dummy"; vars.dummy_state = 0; }
object Host "poc_parent_1" { check_command = "dummy"; vars.dummy_state = 0; }
object Host "poc_child" { check_command = "dummy"; vars.dummy_state = 0;}

object Dependency "parent_0_to_grandparent" {
    child_host_name = "poc_parent_0"
    parent_host_name = "poc_grand_parent"
}

for (i in range(2)) {
    object Dependency "dep-" + i use (i) {
        child_host_name = "poc_child"
        parent_host_name = "poc_parent_" + i
        redundancy_group = "broken_red_deps"
    }
}

Expected behavior

The expected behavior is that poc_child host remains reachable despite of the state of poc_grand_parent.

Screenshots

Your Environment

Include as many relevant details about the environment you experienced the problem in

Version used (icinga2 --version):

$ /usr/local/icinga2/sbin/icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.2-1)

Copyright (c) 2012-2024 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Red Hat Enterprise Linux
  Platform version: 8.9 (Ootpa)
  Kernel: Linux
  Kernel version: 4.18.0-513.11.1.el8_9.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 8.5.0
  Build host: ol8-template.localdomain
  OpenSSL version: OpenSSL 1.1.1k  FIPS 25 Mar 2021

Application information:

General paths:
  Config directory: /usr/local/icinga2/etc/icinga2
  Data directory: /usr/local/icinga2/var/lib/icinga2
  Log directory: /usr/local/icinga2/var/log/icinga2
  Cache directory: /usr/local/icinga2/var/cache/icinga2
  Spool directory: /usr/local/icinga2/var/spool/icinga2
  Run directory: /usr/local/icinga2/var/run/icinga2

Old paths (deprecated):
  Installation root: /usr/local/icinga2
  Sysconf directory: /usr/local/icinga2/etc
  Run directory (base): /usr/local/icinga2/var/run
  Local state directory: /usr/local/icinga2/var

Internal paths:
  Package data directory: /usr/local/icinga2/share/icinga2
  State path: /usr/local/icinga2/var/lib/icinga2/icinga2.state
  Modified attributes path: /usr/local/icinga2/var/lib/icinga2/modified-attributes.conf
  Objects path: /usr/local/icinga2/var/cache/icinga2/icinga2.debug
  Vars path: /usr/local/icinga2/var/cache/icinga2/icinga2.vars
  PID path: /usr/local/icinga2/var/run/icinga2/icinga2.pid

Operating System and version:

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.9 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.9 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

Enabled features (icinga2 feature list):

# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite ido-mysql ido-pgsql influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker icingadb influxdb mainlog notification

Icinga Web 2 version and modules (System - About):

The text was updated successfully, but these errors were encountered:

Al2Klimov · 2024-04-25T11:40:39Z

Hello Jose!

Does just Web mis-indicate the reachability or the Icinga 2 API, too?

Best,
A/K

jjuanino · 2024-04-25T20:11:07Z

Hi Alexander,

in the icinga2 console I get the following (output snipped):

<1> => get_host("poc_child")
{
	__name = "poc_child"
	check_attempt = 1.000000
	check_command = "dummy"
	check_interval = 300.000000
	last_check_result = {
		active = true
		command = "dummy"
		exit_status = 0.000000
		output = "Check was successful."
		previous_hard_state = 99.000000
		vars_after = {
			attempt = 1.000000
			reachable = false    ◄◄◄◄◄◄◄◄◄◄◄◄◄◄◄◄
			state = 0.000000
			state_type = 1.000000
		}
		vars_before = {
			attempt = 1.000000
			reachable = true
			state = 0.000000
			state_type = 1.000000
		}
	}
	last_reachable = false   ◄◄◄◄◄◄◄◄◄◄◄◄◄◄◄◄
}

Best regards

Al2Klimov · 2024-05-28T14:46:06Z

The issue is as follows: when poc_grand_parent becomes down, poc_parent_0 becomes unreachable (as usual), but poc_child also, which is unexpected.

poc_child indeed seems to misbehave, but only after yet another check of itself after poc_grand_parent is down.

jjuanino · 2024-05-30T17:57:36Z

Yes, that is right, you have to check the services several times to reproduce the issue. The test presented is a bit contrived to show the behavior, but in the real world you get the issue in a more natural way. Regards.

nilmerg · 2024-08-30T07:56:16Z

I'm also able to reproduce this with just a single check now on the child.

nilmerg · 2024-10-16T10:13:11Z

Just checked the code to understand the behavior, but for another reason, and noticed the cause for this issue. Checkable::IsReachable checks whether any parent is unreachable before considering any redundancy groups. Redundancy groups only apply if all parents are reachable.

Al2Klimov added the bug Something isn't working label Jun 18, 2024

nilmerg mentioned this issue Oct 16, 2024

Let redundancy groups not just fail #10190

Open

yhabteab self-assigned this Nov 12, 2024

yhabteab added the area/runtime Downtimes, comments, dependencies, events label Nov 12, 2024

yhabteab linked a pull request Nov 12, 2024 that will close this issue

Checkable: Don't skip redundancy group checks for parent dependencies #10228

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Child host becomes unreachable in redundant group when parent is unreachable #10014

Child host becomes unreachable in redundant group when parent is unreachable #10014

jjuanino commented Mar 3, 2024

Al2Klimov commented Apr 25, 2024

jjuanino commented Apr 25, 2024

Al2Klimov commented May 28, 2024

jjuanino commented May 30, 2024

nilmerg commented Aug 30, 2024

nilmerg commented Oct 16, 2024

Child host becomes unreachable in redundant group when parent is unreachable #10014

Child host becomes unreachable in redundant group when parent is unreachable #10014

Comments

jjuanino commented Mar 3, 2024

Describe the bug

To Reproduce

Expected behavior

Screenshots

Your Environment

Al2Klimov commented Apr 25, 2024

jjuanino commented Apr 25, 2024

Al2Klimov commented May 28, 2024

jjuanino commented May 30, 2024

nilmerg commented Aug 30, 2024

nilmerg commented Oct 16, 2024