Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

baremetal: improve debuggability of ipi deployments #328

Closed
wants to merge 6 commits into from
Closed

baremetal: improve debuggability of ipi deployments #328

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a3bf88f
baremetal: improve debuggability of ipi deployments
stbenjam May 15, 2020
45ad3cd
Update enhancements/baremetal/debuggability-of-baremetal-ipi-deployme…
stbenjam May 19, 2020
0b73120
Apply suggestions from code review
stbenjam May 19, 2020
8b05817
Add more see-also references, and other clean-ups
stbenjam May 19, 2020
1440b6f
Clarify worker deployments should make operator degraded
stbenjam May 19, 2020
545d738
Address comments
stbenjam May 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
256 changes: 256 additions & 0 deletions enhancements/baremetal/debuggability-of-baremetal-ipi-deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,256 @@
---
title: debuggability-of-baremetal-ipi-deployment
authors:
- "@stbenjam"
reviewers:
- "@abhinavdahiya"
- "@dtantsur"
- "@enxebre"
- "@hardys"
- "@juliakreger"
- "@markmc"
- "@sadasu"
approvers:
- TBD
creation-date: 2020-05-15
last-updated: 2020-05-15
status: provisional
see-also:
- https://github.com/openshift/enhancements/pull/212
- https://github.com/openshift/installer/issues/2009
- https://github.com/openshift/installer/issues/2569
- https://github.com/openshift/installer/pull/3535
- https://storyboard.openstack.org/#!/story/2007664
replaces:
superseded-by:
---

# Improve debuggability of baremetal IPI deployment failures

## Release Signoff Checklist

- [ ] Enhancement is `implementable`
- [ ] Design details are appropriately documented from clear requirements
- [ ] Test plan is defined
- [ ] Graduation criteria for dev preview, tech preview, GA
- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)

## Open Questions

1. Should the installer error or warn when compute replicas quantity is
not met, even if at lest 2 workers deployed (enough to get a
functional cluster)?
2. In order to bubble up information about worker failures to the
installer, the most likely solution to me seems to use operator
status as Degraded, with a relevant error message. Could we mark
machine-api-operator when workers fail to roll out?

## Summary

In OpenShift 4.5, we improved the existing installer validations for
baremetal IPI to identify early problems. Those include identifying
duplicate baremetal host records, insufficient hardware resources to
deploy the requested cluster size, reachability of RHCOS images, and
networking misconfiguration such as overlapping networks or DNS
misconfiguration.

However, a variety of situations exist where deployments fail for
reasons that were not preventable during the pre-install validations.
These failures in baremetal IPI are hard to diagnose. Errors from
baremetal-operator and ironic are often not presented to the user, and
even when they are the installer doesn't provide context about what
action to take.

This enhancement request is a broad attempt at categorizing the types of
deployment failures, and what information we could present to the user
to make identifying root causes easier.

## Motivation

The goal of this enhancement is to improve the day 1 install experience
and reduce the perception of complexity in baremetal IPI deployments.

### Goals

- Any deployment that ends in an unsuccessful install must provide the
user clear and actionable information to diagnose the problem.

### Non-Goals

- Addressing the underlying causes of the failures is not the goal of
this enhancement.

## Proposal

Broadly, deployments fail due to problems encountered during these
installation activities:

- Pre-bootstrap (image downloading, manifest creation, etc)
- Infrastructure automation (Terraform)
- Bootstrap
- Bare Metal Host Provisioning (Control Plane and Workers)
russellb marked this conversation as resolved.
Show resolved Hide resolved
- Operator Deployment (i.e., those rolled out by CVO)

We believe that since 4.5, pre-bootstrap errors are usually detected,
and useful information is presented to the user about how to rectify the
problem, so this enhancement request will focus on failures that occur
from terraform onward.

### Kinds of deployment failures

#### Infrastructure Automation (Terraform)

Bare metal IPI relies on terraform to provision a libvirt bootstrap
virtual machine and the bare metal control plane hosts. We use
`terraform-provider-libvirt` and `terraform-provider-ironic` to
accomplish those goals.

Both providers report failures when encountered, but there's usually
little information provided to the user about what to do in the
OpenShift context. Given a specific terraform error, the installer
should provide specific information to the user about how to continue
troubleshooting.

#### Bootstrap Failures

The bootstrap runs a couple of baremetal-specific services, including
stbenjam marked this conversation as resolved.
Show resolved Hide resolved
Ironic and a utility that populates hardware details for the control
plane hosts.

Bootstrap typically fails for baremetal when we can't download the
machine-os image into our local HTTP cache. Less common, but still
sometimes seen are that services such as dnsmasq, mariadb, ironic-api,
ironic-conductor, or ironic-inspector fail.

Failures on bootstrap services rarely result in any indication to the
user that something went wrong other than that there was a timeout.

The installer has a feature for log gathering on bootstrap failure that
does not work on baremetal. This should be the first priority, but even
in this case a user still needs to look into an archive containing many
logs to identify a failure.

Ideally there would be some mechanism to identify and extract useful
information and display it to the user.
Comment on lines +118 to +124
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

openshift/installer#2569
^^ already looking at making these problems more easy to report in the long term.

For now the installer now has list of common failures and how to identity them in https://github.com/openshift/installer/blob/master/docs/user/troubleshootingbootstrap.md#common-failures
the goal is to curate a list of detectable failures and then automatically do it as part of analysis.

the initial approach in 2569 was that, you show users most failure logs from the bundle and let them decide for themselves, but personally i would like us to come up with common known failure list and then just show this was the error, and here's how you might resolve this.


#### Bare Metal Host Provisioning

Whether the control plane or worker nodes, provisioning of bare metal
hosts can fail in the same ways, although the communication path to
provide feedback is different in each case. For the control plane,
information about failure is presented to the user via terraform. For
workers, it is currently only shown on the `BareMetalHost` resource or
by examining baremetal-operator logs. Failure to deploy a worker should
be reflected by marking either the `machine-api-operator` or the future
`cluster-baremetal-operator` degraded.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to make sure this behavior is similar to behavior for other platforms. Should the MAO or CBO be marked degraded when 1 worker fails to deploy? What if that is an issue with the worker itself? How can we distinguish between errors in the resource being provisioned verses an issue in the control plane?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are all very good questions! It would be helpful to understand from someone in MAO (maybe @enxebre) how it works for other platforms, but my understanding is MAO doesn't go degraded from this case.

I think that MAO should show degraded if replicas are not met, or at the very least, if replicas are < 2, since we know we need 2 to get a working cluster on day 1 (unless controlplane is scheduable).

Perhaps cluster-operator-baremetal should also go degraded if provisioning fails, with more specific error messages bubbled up from baremetal-operator/ironic.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a discussion with the MAO team this is what I learnt. The MAO would go into a degraded state only when the pods that it is responsible for deploying, fail to come up. When a resource it manages, in this case a worker Machine, does not come up, the MAO does not go into a failed/degraded state for other platforms and should probably be the same for baremetal too.
When the initial deployment with 2 workers fails, then it should be considered an Installer error and we should provide the best/detailed errors message we can provide by bubbling up what we can get from BMO and/or Ironic. The MAO team believes that this is not a reason to put the operator in a degraded state.
Since the baremetal platform is special, we could come up with a semantic on day 2 where if we notice a large number of worker failures (for example 90% of workers are not coming up), then the aggregated bad state could result in the operator being put in the degraded state. Currently, the operator does not have an aggregated view, so that needs to be added to the SLO at a future time.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a discussion with the MAO team this is what I learnt. The MAO would go into a degraded state only when the pods that it is responsible for deploying, fail to come up. When a resource it manages, in this case a worker Machine, does not come up, the MAO does not go into a failed/degraded state for other platforms and should probably be the same for baremetal too.

Yea, it doesn't go to a degraded state for anyone and I think that's a mistake and what I'm proposing to change here.

When the initial deployment with 2 workers fails, then it should be considered an Installer error and we should provide the best/detailed errors message we can provide by bubbling up what we can get from BMO and/or Ironic. The MAO team believes that this is not a reason to put the operator in a degraded state.

The main way the installer gets information about deployment success is largely through operator states via CVO, it would need a special case to count workers meeting the requested number of replicas.

If MAO can't provision machines, why is that not a degraded state of the operator?

CC: @abhinavdahiya

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stbenjam This docmentation https://github.com/openshift/cluster-version-operator/blob/master/docs/dev/clusteroperator.md#what-should-an-operator-report-with-clusteroperator-custom-resource helped me understand the semantics behind different Operator states.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Failure to deploy a worker should
be reflected by marking either the machine-api-operator or the future
cluster-baremetal-operator degraded.

There are countless transient scenarios for "Failure to deploy a worker". This makes impractical putting a reasonable generic semantic on top of it. And so this makes worthless to let the overall operator going degraded in such a heterogeneous scenario.

Instead I believe the boundaries to signal the details of theses errors belong to individual machine resource conditions and any lower level resource. Just like we do for any other provider https://github.com/openshift/cluster-api-provider-aws/blob/master/pkg/apis/awsprovider/v1beta1/awsproviderstatus_types.go#L40

Then to communicate "Failure to deploy a worker" We already trigger alerts any time a machine has no node regardless of the failure details and regardless the provider. So each failure details can then be analysed in the format described above.

Beyond all the above, regardless of the failure details and based on the overall health of the cluster (e.g 99 out 100 machines has no node) we might decide our criteria for a semantic that represents a permanent global issue and choose to let the mao going degraded in that case. But that's a separate scoped discussion.

Copy link
Member Author

@stbenjam stbenjam Jul 16, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already trigger alerts any time a machine has no node regardless of the failure details and regardless the provider.

These don't show up in the installer output, and as far as I know they do try to capture alerts.

If you've ever done an install and ended up with a non-viable cluster due to insufficient workers, the UX is unacceptable. You get a report of a dozen failing operators and absolutely no indication it's because you don't have enough workers. @sdodson previously mentioned maybe we could do something in the installer about it (#328 (comment)), which may help the problem I guess, but doesn't feel like the right solution to me as machine-api-operator, being the top-level operator for dealing with machines, should be signaling clearly about the problem.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These don't show up in the installer output, and as far as I know they do try to capture alerts.

That'd be then a very a specific issue: "Installer output not capturing some existing alerts as expected".

ended up with a non-viable cluster due to insufficient workers,

I agree. That scenario should be covered by my last statement in the previous comment.


Provisioning can fail in many ways. The most difficult to troubleshoot
are simply when we fail to hear back from a host. Buggy UEFI firmware
may prevent PXE, a kernel could panic, or even a network cable may be
unplugged. In these cases, we should inform the user what little
information Ironic was able to discern, but also provide a suggestion
that the most effective way to troubleshoot the problem is examination
of the console of the host.

stbenjam marked this conversation as resolved.
Show resolved Hide resolved
An infrequent, but possible outcome of deployment to bare metal hosts,
is that Ironic is successful in cleaning, inspecting, and deploying a
host. After Ironic lays down an image on disk and reboots, Ironic marks
the host ‘active’. However, when the host boots again it’s possible that
there’s a catastrophic problem such as a kernel panic or fail to
configure with ignition. From Ironic's perspective, it's done it's duty,
and is unaware the host failed to come up. The feedback to the user is
only that there was a timeout.

#### Operator Deployment

Operator deployment failures are rarely platform-specific, although
there is one case that should be addressed. When worker deployment
fails, possibly due to provisioning issues like those described above, a
variety of operators may report failures such as ingress, console, and
others that cannot run on the control plane.

When this happens, the installer times out, reports to the user a large
number of operators failed to roll out, and no useful context about what
to do or why the operators failed.
Comment on lines +162 to +164
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's on the operator owners to make sure the errors are clear. Think about how it is not only installer that is the consumer of these message, but also admins during upgrades.

So personally the goal should be to ensure that each operator is responsible for using clear error messages in the status.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, it's just on day 1 the worker deployment failure seems to be special to me. It causes a lot of noise as a bunch of operators start reporting error messages that make it hard to point to a root cause unless you've seen the problem before. I don't think machine-api operator even reports anything useful when this happens, but if it did, it'd get lost in mix of the many other failing operators.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we take ingress/console operator as an example - if the worker fails, they will be in error - but, from my experience of installing openshift for first few times - the user will have no idea that this is the reason. He will just see that those operators are down.
What is possible maybe to do is to have some kind of 'validators' - either from the installer binary or as an operator - that can analyze logs or cluster runtime state (with minimal requirement for cluster functionality - such as passwordless ssh between nodes) that can look into the state of the cluster and explain the user what went wrong. If we provide an infra for writing those validators, then operators owners / qe / intergration team will be able to enhance those once they rn into an issue that it was hard to analyze.

Copy link
Member

@sdodson sdodson May 19, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they asked for a certain number of works and they didn't get them that seems reasonable to have a special error for that.

I also think some generic orientation regarding how to investigate operators failing may help as well. They'll need to learn that skill eventually no matter what. So documenting how to look at an Operator's status and referencing that seems to be something worth doing no matter what. Ingress for example often tells you that the dns entry doesn't exist but people don't even know where to look for that.


#### User Stories

##### Show more information from terraform

- As a user, I want terraform to report last_error and status from
ironic in case of deployment failure.

- As a user, I want the installer to provide suggestions for causes
of failure. See the existing work for translating terraform error
messages that is being done in https://github.com/openshift/installer/pull/3535.

#### Extract relevant logs from the bootstrap

- As a user, I would like the installer to extract and display error
messages from bootstrap journal when relevant errors can be
identified.

#### Implement bootstrap gather

- As a user, I want the installer to automatically gather logs when
bootstrap fails on the baremetal IPI platform, like it does for other
platforms.

See also:
- https://github.com/openshift/installer/issues/2009

#### Show errors from machine controllers

- As a user, I want the installer logs to bubble information up from
either machine-api-operator or cluster-baremetal-operator about why
workers failed to deploy. These operators should be degraded when
machine provisioning fails.

#### Callback to Metal3

- As a user, I want my host to callback to Metal3/Ironic from ignition
when RHCOS boots.

See also:
- https://storyboard.openstack.org/#!/story/2007664

### Implementation Details/Notes/Constraints

### Risks and Mitigations

Some stories may impact the design of software managed by teams other
than the baremetal IPI team. These including the installer and
machine-api-operator teams, for example.

## Design Details

### Test Plan

**Note:** *Section not required until targeted at a release.*

### Graduation Criteria

**Note:** *Section not required until targeted at a release.*

### Upgrade / Downgrade Strategy

This enhancement largely consists of day 1 considerations, however we
are suggesting that worker deployment failures be reflected as a
Degraded operator status either in `machine-api-operator` or
`cluster-baremetal-operator`. This would prevent an upgrade and is an
intentional change in behavior.

### Version Skew Strategy

As these are day 1 considerations for greenfield deployments, no version
skew strategy is needed.

## Implementation History

## Drawbacks

## Alternatives

An alternative approach would be to provide troubleshooting
documentation and leave users to uncover the root causes of failures on
their own, which is largely what happens today.