Change automate methods to communicate via REST API

[miq-bot]

Much of this was discussed at the ManageIQ design summit for Botvinnik. However, @mkanoor and I recently sat down and brainstormed our way through many of the larger problems, and we think we've come to a cleaner design.

There may be more things I'm not thinking of, but I just wanted to start getting this information out there. Let's keep this document updated as we find more issues and/or come up with more ways to solve them.

---

Automate currently runs methods by

• starting a DRb server
• spawning the ruby method wrapped in a preamble and postamble, with the preamble starting a DRb client
• client uses service models and helper classes like $evm to communicate with the server

This causes a number of problems and headaches, which may be able to be addressed by moving to a REST API based implementation. Some notable issues:

• DRb requires server side object management so that the server doesn't garbage collect objects in use by the client. This is a huge headache which will just go away with REST.

• Every time we add a new object or method we must manually expose it via automate. This is prone to error. This would instead become an exercise in exposing via the REST API, which in nearly all cases should be done anyway.

• DRb is used to handle interactions with the automate workspace, which is in the server's memory, from the client. This can be handled by dumping the workspace into some format like JSON or YAML, and passing it on STDIN into the launched process.

  The automate method preamble would take STDIN and hold it in memory, lazy loading it on first access. If the method needs to modify the workspace, then it can be modified. $evm would no longer be special and can just be a simple object defined in the preamble that has the knowledge of manipulating the workspace in this manner. $evm.log could even just write directly to the log in question or could write to STDOUT.

  In the postamble we will dump the workspace to STDOUT. In order to prevent collision with user usage of STDOUT, we can prefix the dump with some sort of delimiter (e.g. "=" * 80). STDERR can be used to transfer any exception, or perhaps that becomes part of the workspace that's dumped into STDOUT. Again, a delimiter might be necessary.

  The server, when it detects that the child process has completed, will read from the STDOUT of the child process, find the last delimiter, and process the workspace back into memory, if changed.

  There may be optimizations here to prevent unnecessarily dumping and loading the workspace.

• DRb is also used for database interaction. The database interaction that is currently done via service models, can be done completely through the REST API. A Ruby gem needs to be created from the cfme_client code, which additionally has more of an OO design. (I'll call it manageiq_client for reference). Once that's available, then the $evm.vmdb can just be a front for accessing those objects from the manageiq_client gem. We could even deprecate $evm.vmdb in favor of having the automate methods use the normal manageiq_client gem directly.

  For example, currently $evm.vmdb["vm", 21] returns a MiqAeServiceVmVmware instance for VM 21. If the REST API gem had it's own OO based types, I could see $evm.vmdb["vm", 21] instead calling the manageiq_client gem, ultimately returning a ManageiqClient::Vm instance for VM 21

  With this in place, I think the entirety of the service models just goes away

• DRb is also used for built-in methods (like send_email). This can be handled by exposing them from the REST API, and passing the dumped workspace from the method as a parameter to the REST API. These may need some sort of system-only access rules, but then again, maybe not.

Other thoughts:

• With database interaction via the REST API, and the workspace passed via STDIN/STDOUT in a common format like JSON or YAML, then the support for other languages opens up. These formats are widely supported, so they could be passed to any kind of method whether it be Ruby, Python, Shell, etc. All that would be needed from the server is possibly some preamble/postamble code in the native language to simplify setting up helper objects and catching errors.

---

TODO

• Other parts of $evm object that we haven't even looked at like get/set_state_var, instance_*, etc
• What user would automate methods run as if they are accessing the REST API. For the system things like the built-in methods, it's some system account, but what about other methods? Can they just be system too?

cc @gmcculloug @mkanoor @tinaafitz @chessbyte
cc @abellotti (re: REST API stuff and cfme_client split out into manageiq_client gem)
cc @kbrock (re: system-only access rules)

---

This issue was moved to this repository from ManageIQ/manageiq#2215, originally opened by @Fryguy

[kbrock]

The preamble is currently cut and pasted into a customer's scripts.
Alternatively, it could be implemented as a parent class.

What user would automate methods run as

We (always?) pass the requesting/effective user to automate. This is needed for tenancy.

One thing to keep in mind. When a user does not have privileges to perform an operation, the user makes a request from automate to perform the operation on their behalf. So it is essentially a mechanism to escalated privileges.

So ignoring the fact that we don't have the user's password, we also don't properly establish the escalated user's identity when running automate methods.

UPDATE: not positive this is the intent of automate.

[Fryguy]

The preamble is currently cut and pasted into a customer's scripts.
Alternatively, it could be implemented as a parent class.

Oh that's really interesting.

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 6 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions!

[mkanoor]

For Ansible Playbooks we have enhanced the REST-API to allow access to the Automate Workspace.
https://www.manageiq.org/docs/reference/latest/api/reference/automate_workspaces

The engine creates a token and passes the token to the Ansible Playbook as an extra var which can be used by the playbook to connect to our REST API and access the workspace, update the workspace and also pass back updates to the workspace.

[kbrock]

Overview

When an automate method is called, a workspace ( MiqAeWorkspaceRuntime) is created and it is wrapped by an MiqAeService object. This is stored in an in-memory hash in the server process.

drb spawns a child process and the child process obtains a drb reference to the MiqAeService instance (living in server memory) and stores it in the $evm variable. The MiqAeService instance provides the child process with access to manageiq database and services.

Again: All manageiq provided services are provided from within the server process, accessible via $evm over drb.

Goal

Getting away from relying upon drb needs 2 steps:

1. Access differently: Convert $evm to be a client side object
2. Spawn Differently: Kick off automate scripts with something other than drb.

Benefits:

1. Remove the need for a separate Automate Service Model API.
2. Enhance the existing ReST API to provide more functionality to all
3. ReST is more container friendly than drb.
4. Remove drb which fixes some memory peculiarities.
5. HTTP allows an easier worker pool model than our current spawning a process for every automate request.

Access

The client will be able to perform some functionality without the server. These methods will be moved into the child process.
Other functions require the server. Some of these include sending mail, or a typical database access layer.

function	new approach	comment
exit
raise		errors that are raised
enable_rbac	unneeded
disconnect_sql	unneeded
log()	keep, simplified. $stdout, $stderr too
get_state_var	helper into workspace?
ansible_stats_vars

Concepts needed:

• tracking_label
• workspace: is this in client or server memory?
• current[service_object]

Spawn

We currently have a number of ways of invoking automate.

1. From automate (remote task, local task)
2. Inline to populate dialogs
3. an external event (like Vm creation) triggers an event

Also there are a number of templates to invoke:

1. ansible
2. automate