Which data should be processed from ibquerryerrors?

[gabrieleiannetti]

Hi,

the exporter processes the switch data retrieved from ibqueryerrors.

We get SymbolErrorCounter errors from ibqueryerrors, when executed without --switch parameter.

As we can see, those errors are increasing within next day...

13.04.2021:
$ ibqueryerrors > ibqueryerrors.txt

$ grep "SymbolErrorCounter" ibqueryerrors.txt
GUID 0xc42a10300dcf8e2 port 1: [SymbolErrorCounter == 3] [PortXmitWait == 12097916]
GUID 0xc42a10300dd0bea port 1: [SymbolErrorCounter == 4] [PortXmitWait == 224462059]

14.04.2021:
$ ibqueryerrors > ibqueryerrors.txt

$ grep "SymbolErrorCounter" ibqueryerrors.txt

   GUID 0xc42a10300dcf91a port 1: [SymbolErrorCounter == 4] [PortXmitWait == 36939]
   GUID 0xc42a10300dcf8e2 port 1: [SymbolErrorCounter == 7] [PortXmitWait == 58359075]
   GUID 0xc42a10300dd0bea port 1: [SymbolErrorCounter == 4] [PortXmitWait == 612313936]

Are these errors relevant?

I do not understand, why they do not show up at the switch data:

$ ibqueryerrors --switch > ibqueryerrors_switch_data.txt

$ grep "SymbolErrorCounter" ibqueryerrors_switch_data.txt

I also checked the file ibqueryerrors_switch_data.txt for the GUIDs listed above,
but cannot find any entries with an exact match:

grep "0xc42a10300dd0bea" ibqueryerrors_switch_data.txt`

But I can find very similar GUIDs in the switch data (with an additional 0 in the address after the 0x):

$ grep "c42a10300dcf91a" ibqueryerrors_switch_data.txt 
       Link info:   1309  15[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>  0x0c42a10300dcf91a   1291    1[  ] "XXX0906 mlx5_0" ( )

$ grep "c42a10300dcf8e2" ibqueryerrors_switch_data.txt 
       Link info:   1288  15[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>  0x0c42a10300dcf8e2   1284    1[  ] "XXX0925 mlx5_0" ( )

$ grep "c42a10300dd0bea" ibqueryerrors_switch_data.txt 
       Link info:   1180  10[  ] ==( 4X       14.0625 Gbps Active/  LinkUp)==>  0x0c42a10300dd0bea   1243    1[  ] "XXX0796 mlx5_0" ( )

Any ideas?

[guilbaults]

It's probably a error on the server HCA, you can check those errors with perfquery on each server. You can use ibhosts to figure out the hostname of the server based on the GUID.

I initially decided to use --switch because IO counters of each switches will also get the traffic from the servers, but I forgot about errors that are only on the counters on the servers.

# perfquery -x
# Port extended counters: Lid 1014 port 1 (CapMask: 0x5A00 CapMask2: 0x0000000)
PortSelect:......................1
CounterSelect:...................0x0000
PortXmitData:....................2669532118478
PortRcvData:.....................4146954291622
PortXmitPkts:....................10098214319
PortRcvPkts:.....................11499100001
PortUnicastXmitPkts:.............10098212193
PortUnicastRcvPkts:..............11498803983
PortMulticastXmitPkts:...........2126
PortMulticastRcvPkts:............296018
CounterSelect2:..................0x00000000
SymbolErrorCounter:..............0
LinkErrorRecoveryCounter:........0
LinkDownedCounter:...............0
PortRcvErrors:...................0
PortRcvRemotePhysicalErrors:.....0
PortRcvSwitchRelayErrors:........0
PortXmitDiscards:................0
PortXmitConstraintErrors:........0
PortRcvConstraintErrors:.........0
LocalLinkIntegrityErrors:........0
ExcessiveBufferOverrunErrors:....0
VL15Dropped:.....................0
PortXmitWait:....................0
QP1Dropped:......................0

[gabrieleiannetti]

I have checked the HCAs on the three hosts and I could see there the SymbolErrorCounter.
They have no more positve error counter listed.

Simple running perfquery is fine and is sufficient to check for errors.
Just to say, when running it with -x for showing extended port counters the SymbolErrorCounter is 0 here on all three hosts.

Anyways, currently I have also detected with ibqueryerrors the following errors in our fabric:

• SymbolErrorCounter
• PortRcvRemotePhysicalErrors
• PortLocalPhysicalErrors

It would be helpful, if the the exporter will also export additional errors retrieved by ibqueryerrors:

• Errors for host channel adapter
• Errors on STDERR e.g. "bad status 110; Connection timed out" (query_smp.c:199; Connection timed out #12)

[guilbaults]

If I remove --switch from the ibqueryerrors, I do get some errors on 4 servers that are using older cards connected at QDR instead of EDR in this network. Removing the --switch is probably the easiest way to gather all the HCA stats, but will require a bit more work to handle errors. The alternative would be to run it twice, once with --switch and another one with --ca to gather all the servers.

2021-04-19 19:07:26,747 - ERROR - ibwarn: [117052] query_and_dump: PortXmitDiscardDetails query failed on lustreXX-mds2 mlx4_0, Lid 30 port 1
ibwarn: [117052] query_and_dump: PortXmitDiscardDetails query failed on lustreXX-mds1 mlx4_0, Lid 15 port 1
ibwarn: [117052] query_and_dump: PortXmitDiscardDetails query failed on lustreXX-oss2 mlx4_0, Lid 73 port 1
ibwarn: [117052] query_and_dump: PortRcvErrorDetails query failed on lustreXX-oss1 mlx4_0, Lid 39 port 1
ibwarn: [117052] query_and_dump: PortXmitDiscardDetails query failed on lustreXX-oss1 mlx4_0, Lid 39 port 1

[gabrieleiannetti]

We also see errors on STDERR from ibqueryerrors that indicate collecting metrics for specific CA are failing...

Created an issue for this #32.

[gabrieleiannetti]

Merge request for processing CA and switches submitted #31.