Connection reset by remote peer (switches) in a "random" manner

[lhcunha]

Summary

When I run a playbook that sends commands on remote switches, sometimes some of the tasks fails on some switches, returning an error message "[Errno 104] Connection reset by peer".
Some important facts:

• Im using AWX
• The playbook is scheduled to run every 10min
• The same switch that fails on the previous playbook run can succeed on the next playbook run
• the inventory has aprox. 30 switches, all located in the same subnet
• sometimes, all the switches succeeds, sometimes, a couple switches fails with this message.
• no specific switches failing, could be any them
• Switches: J9773A 2530-24G-PoEP (YA.16.10.0015)

Issue Type

Bug Report

Component Name

arubanetworks.aos_switch.arubaoss_command

Ansible Version

AWX 21.13.0
ansible-core: 2.9

Configuration

I'm using the default ansible configuration. No parameters were updated.

OS / Environment

RHEL 8

Steps to Reproduce

I'm putting one of the tasks from the playbook as an example, but it fails in different tasks in a random way, not on a specific task.

- name: GET SHOW INT BRIEF OUTPUT TO PREVENT FALSE-POSITIVE
  arubanetworks.aos_switch.arubaoss_command:
    commands:
      - "show interface brief"
    wait_for: result[0] contains Status
  register: int_brief_output_prevent

- name: Print show interface brief output
  debug:
    msg: "{{ int_brief_output_prevent }}"

Expected Results

{
  "msg": {
    "changed": false,
    "stdout": [
      "Status and Counters - Port Status\n\n                   | Intrusion                           MDI  Flow\n  Port  Type       | Alert     Enabled Status Mode       Mode Ctrl\n  ----- ---------- + --------- ------- ------ ---------- ---- ----\n  1     100/1000T  | No        Yes     Up     1000FDx    MDI  off \n  2     100/1000T  | No        Yes     Up     10HDx      MDI  off \n  3     100/1000T  | Yes       Yes     Up     10HDx      MDI  off \n  4     100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  5     100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  6     100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  7     100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  8     100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  9     100/1000T  | No        Yes     Up     10HDx      MDI  off \n  10    100/1000T  | No        Yes     Up     10HDx      MDI  off \n  11    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  12    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  13    100/1000T  | No        Yes     Up     100FDx     MDIX off \n  14    100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  15    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  16    100/1000T  | No        Yes     Up     10HDx      MDIX off \n  17    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  18    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  19    100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  20    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  21    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  22    100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  23    100/1000T  | No        Yes     Down   1000FDx    MDI  off \n  24    100/1000T  | No        Yes     Down   1000FDx    MDIX off \n  25    1000LX     | No        Yes     Up     1000FDx    NA   off \n  26               | No        Yes     Down   .               off \n  27               | No        Yes     Down   .               off \n  28               | No        Yes     Down   .               off"
    ],
    "stdout_lines": [
      [
        "Status and Counters - Port Status",
        "",
        "                   | Intrusion                           MDI  Flow",
        "  Port  Type       | Alert     Enabled Status Mode       Mode Ctrl",
        "  ----- ---------- + --------- ------- ------ ---------- ---- ----",
        "  1     100/1000T  | No        Yes     Up     1000FDx    MDI  off ",
        "  2     100/1000T  | No        Yes     Up     10HDx      MDI  off ",
        "  3     100/1000T  | Yes       Yes     Up     10HDx      MDI  off ",
        "  4     100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  5     100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  6     100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  7     100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  8     100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  9     100/1000T  | No        Yes     Up     10HDx      MDI  off ",
        "  10    100/1000T  | No        Yes     Up     10HDx      MDI  off ",
        "  11    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  12    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  13    100/1000T  | No        Yes     Up     100FDx     MDIX off ",
        "  14    100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  15    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  16    100/1000T  | No        Yes     Up     10HDx      MDIX off ",
        "  17    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  18    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  19    100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  20    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  21    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  22    100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  23    100/1000T  | No        Yes     Down   1000FDx    MDI  off ",
        "  24    100/1000T  | No        Yes     Down   1000FDx    MDIX off ",
        "  25    1000LX     | No        Yes     Up     1000FDx    NA   off ",
        "  26               | No        Yes     Down   .               off ",
        "  27               | No        Yes     Down   .               off ",
        "  28               | No        Yes     Down   .               off"
      ]
    ],
    "ansible_facts": {
      "discovered_interpreter_python": "/usr/bin/python"
    },
    "deprecations": [
      {
        "msg": "Distribution rhel 8.7 on host SW-MAG040-ASW04-R02-ADM should use /usr/libexec/platform-python, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information",
        "version": "2.12"
      }
    ],
    "failed": false
  },
  "_ansible_verbose_always": true,
  "_ansible_no_log": false,
  "changed": false
}

Actual Results

{
  "msg": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 200, in send\n    response = recv_data(sf)\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 76, in recv_data\n    d = s.recv(header_len - len(data))\nConnectionResetError: [Errno 104] Connection reset by peer\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/bin/ansible-connection\", line 310, in main\n    conn.set_options(var_options=variables)\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 184, in __rpc__\n    response = self._exec_jsonrpc(name, *args, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 150, in _exec_jsonrpc\n    out = self.send(data)\n  File \"/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py\", line 204, in send\n    raise ConnectionError('unable to connect to socket', err=to_text(e, errors='surrogate_then_replace'), exception=traceback.format_exc())\nansible.module_utils.connection.ConnectionError: unable to connect to socket\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/bin/ansible-connection\", line 350, in <module>\n    main()\n  File \"/usr/local/bin/ansible-connection\", line 313, in main\n    raise ConnectionError('Unable to decode JSON from response set_options. See the debug log for more information.')\nansible.module_utils.connection.ConnectionError: Unable to decode JSON from response set_options. See the debug log for more information.\n",
  "_ansible_no_log": false
}

Code of Conduct