Using ansible-pylibssh produces Unable to decode JSON from response to get_capabilities(). Received 'None'. error

[netexgb]

SUMMARY

When running cisco.ios.ios_command with ansible_pylibssh, I get a JSON RPC decoding error, which I do not get when using paramiko

ISSUE TYPE

• Bug Report

PYLISSH and LIBSSH VERSION

xxxx@xxxx:~$ python3 -m pip install --upgrade ansible-pylibssh
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: ansible-pylibssh in ./.local/lib/python3.8/site-packages (0.4.0)

STEPS TO REPRODUCE

ANSIBLE VERSION:
xxxx@xxxx:~/.ssh$ ansible --version
ansible [core 2.13.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/xxxx/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/xxxx/.ansible/collections
  executable location = /home/xxxx/.local/bin/ansible
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True


PLAYBOOK:
---
- name: show configuration objects
  hosts: router
  gather_facts: no

  tasks:
    - name: Verify Configuration
      cisco.ios.ios_command:
         commands:
           - sh run | s ^ro.*example
           - sh run | i ip pref.*DEF

      # when: inventory_hostname == "xxxx"
      register: item

    - debug:
         var: item.stdout_lines

EXPECTED RESULTS

xxxx@xxxx:/mnt/c/Users/xxxx/Repositories/$ ansible-playbook cisco_config_verify.yml -u xxxx --ask-pass -vvvvvv
ansible-playbook [core 2.13.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/xxxx/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/xxxx/.ansible/collections
  executable location = /home/xxxx/.local/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
SSH password:
setting up inventory plugins
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Parsed /etc/ansible/hosts inventory source with yaml plugin
Loading collection cisco.ios from /home/xxxx/.ansible/collections/ansible_collections/cisco/ios
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
Loading callback plugin default of type stdout, v2.0 from /home/xxxx/.local/lib/python3.8/site-packages/ansible/plugins/callback/default.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: cisco_config_verify.yml *******************************************************************************************************************Positional arguments: cisco_config_verify.yml
verbosity: 6
remote_user: xxxx
connection: smart
timeout: 10
ask_pass: True
become_method: sudo
tags: ('all',)
inventory: ('/etc/ansible/hosts',)
forks: 5
1 plays in cisco_config_verify.yml

PLAY [show configuration objects] *******************************************************************************************************************META: ran handlers
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
Loading collection ansible.netcommon from /home/xxxx/.ansible/collections/ansible_collections/ansible/netcommon

TASK [Verify Configuration] *************************************************************************************************************************task path: /mnt/c/Users/xxxx/Repositories/xxxx/cisco_config_verify.yml:7
redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<xxxx> attempting to start connection
<xxxx> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /home/xxxx/.local/bin/ansible-connection
<xxxx> local domain socket does not exist, starting it
<xxxx> control socket path is /home/xxxx/.ansible/pc/667f5f96cf
<xxxx> Loading collection ansible.netcommon from /home/xxxx/.ansible/collections/ansible_collections/ansible/netcommon
<xxxx> redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
<xxxxt> Loading collection cisco.ios from /home/xxxx/.ansible/collections/ansible_collections/cisco/ios
<xxxx> redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<xxxx> local domain socket listeners started successfully
<xxxx> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os ios
<xxxx> ssh type is set to auto
<xxxx> autodetecting ssh_type
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
<xxxx> ssh type is now set to paramiko
<xxxx>
<xxxx> local domain socket path is /home/xxxx/.ansible/pc/667f5f96cf
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
<xxxx> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<xxxx> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_command  at /home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/modules/ios_command.py
<xxxx> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_command
<xxxx> ANSIBLE_NETWORK_IMPORT_MODULES: complete

TASK [debug] ****************************************************************************************************************************************task path: /mnt/c/Users/xxxx/Repositories/xxxx/cisco_config_verify.yml:16
redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<xxxx> attempting to start connection
<xxxx> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /home/xxxx/.local/bin/ansible-connection
<xxxx> found existing local domain socket, using it!
<xxxx> invoked shell using ssh_type: paramiko
<xxxxt> ssh connection done, setting terminal
<xxxx> loaded terminal plugin for network_os ios
<xxxx> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<xxxx> firing event: on_open_shell()
<xxxx> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<xxxx> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<xxxx> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<xxxx> ssh connection has completed successfully
<xxxx> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<xxxx> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<xxxx> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<xxxx> updating play_context for connection
<xxxx>
<xxxx> local domain socket path is /home/xxxx/.ansible/pc/667f5f96cf

META: ran handlers
META: ran handlers

PLAY RECAP ******************************************************************************************************************************************xxxx : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

ACTUAL RESULTS

xxxx@xxxx:/mnt/c/Users/xxxx/Repositories/xxxx$ ansible-playbook cisco_config_verify.yml -u xxxx --ask-pass -vvvvvv
ansible-playbook [core 2.13.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/xxxx/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/xxxx/.ansible/collections
  executable location = /home/xxxx/.local/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
SSH password:
setting up inventory plugins
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Parsed /etc/ansible/hosts inventory source with yaml plugin
Loading collection cisco.ios from /home/xxxx/.ansible/collections/ansible_collections/cisco/ios
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
Loading callback plugin default of type stdout, v2.0 from /home/xxxx/.local/lib/python3.8/site-packages/ansible/plugins/callback/default.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: cisco_config_verify.yml *******************************************************************************************************************Positional arguments: cisco_config_verify.yml
verbosity: 6
remote_user: xxxx
connection: smart
timeout: 10
ask_pass: True
become_method: sudo
tags: ('all',)
inventory: ('/etc/ansible/hosts',)
forks: 5
1 plays in cisco_config_verify.yml

PLAY [show configuration objects] *******************************************************************************************************************META: ran handlers
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
Loading collection ansible.netcommon from /home/xxxx/.ansible/collections/ansible_collections/ansible/netcommon

TASK [Verify Configuration] *************************************************************************************************************************task path: /mnt/c/Users/xxxx/Repositories/xxxx/cisco_config_verify.yml:7
redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<xxxx> attempting to start connection
<xxxx> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /home/xxxx/.local/bin/ansible-connection
<xxxx> local domain socket does not exist, starting it
<xxxx> control socket path is /home/xxxx/.ansible/pc/4a1c10f12f
<xxxx> Loading collection ansible.netcommon from /home/xxxx/.ansible/collections/ansible_collections/ansible/netcommon
<xxxx> redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
<xxxx> Loading collection cisco.ios from /home/xxxx/.ansible/collections/ansible_collections/cisco/ios
<xxxx> redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<xxxx> local domain socket listeners started successfully
<xxxx> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os ios
<xxxx> ssh type is set to auto
<xxxx> autodetecting ssh_type
<xxxx> ssh type is now set to libssh
<xxxx>
<xxxx> local domain socket path is /home/xxxx/.ansible/pc/4a1c10f12f
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
<xxxx> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<xxxx ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_command  at /home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/modules/ios_command.py
<xxxx> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_command
<xxxx> ANSIBLE_NETWORK_IMPORT_MODULES: complete
<xxxx> ANSIBLE_NETWORK_IMPORT_MODULES: Result: {'failed': True, 'msg': "Unable to decode JSON from response to get_capabilities(). Received 'None'.", 'exception': '  File "/home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/module_utils/network/ios/ios.py", line 97, in get_capabilities\n    capabilities = Connection(module._socket_path).get_capabilities()\n  File "/home/xxxx/.local/lib/python3.8/site-packages/ansible/module_utils/connection.py", line 194, in __rpc__\n    response = self._exec_jsonrpc(name, *args, **kwargs)\n  File "/home/xxxx/.local/lib/python3.8/site-packages/ansible/module_utils/connection.py", line 173, in _exec_jsonrpc\n    raise ConnectionError(\n', 'invocation': {'module_args': {'commands': ['sh run | s ^ro.*example', 'sh run | i ip pref.*DEF'], 'match': 'all', 'retries': 10, 'interval': 1, 'wait_for': None, 'provider': None}}, '_ansible_parsed': True}
The full traceback is:
  File "/home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/module_utils/network/ios/ios.py", line 97, in get_capabilities
    capabilities = Connection(module._socket_path).get_capabilities()
  File "/home/xxxx/.local/lib/python3.8/site-packages/ansible/module_utils/connection.py", line 194, in __rpc__
    response = self._exec_jsonrpc(name, *args, **kwargs)
  File "/home/xxxx/.local/lib/python3.8/site-packages/ansible/module_utils/connection.py", line 173, in _exec_jsonrpc
    raise ConnectionError(
fatal: [xxxx]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "commands": [
                "sh run | s ^ro.*example",
                "sh run | i ip pref.*DEF"
            ],
            "interval": 1,
            "match": "all",
            "provider": null,
            "retries": 10,
            "wait_for": null
        }
    },
    "msg": "Unable to decode JSON from response to get_capabilities(). Received 'None'."
}

PLAY RECAP ******************************************************************************************************************************************xxxx : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

[webknjaz]

@netexgb have you already reported this to the cisco.ios collection tracker? This might be their bug.

[netexgb]

Hi @webknjaz, I have opened #642 under cisco.ios as a bug. Thanks!

[webknjaz]

Ref ansible-collections/cisco.ios#642

[ng-bsy]

@webknjaz @netexgb
Same thing happens with ansible_network_os=community.network.exos and using ansible.netcommon.net_get since switching from paramiko to ansible-pylibssh 1.1.0
So it's not Cisco-specific

[webknjaz]

@Jakuje could you check if any of your PRs address this?

[Jakuje]

Hard to say. These log do not show if the scp was involved, what is going under the hood, which triggerd me to write the #597 initially (but it looks like it does not add the logging to the same channel as ansible prints, I think). But it would be certainly worth retrying with the updated version!