ARK backup failed with efs provisioner

[pmquang]

I used efs provisioner for creating PV for PO and ark can not backup PV.

This is yaml file to create nginx with efs pv.

#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-example-efs
  labels:
    app: nginx-example-efs

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-logs
  namespace: nginx-example-efs
  labels:
    app: nginx-example-efs
spec:
  storageClassName: aws-efs-2
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: nginx-example-efs
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx-example-efs
    spec:
      volumes:
        - name: nginx-logs
          persistentVolumeClaim:
           claimName: nginx-logs
      containers:
      - image: nginx:1.7.9
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: "/var/log/nginx"
            name: nginx-logs
            readOnly: false
      tolerations:
        - key: "type"
          effect: "NoSchedule"
          value: "MEM"

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-example-efs
  name: my-nginx
  namespace: nginx-example-efs
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx-example-efs
  type: ClusterIP

Log:

Name:         nginx-example-efs
Namespace:    heptio-ark
Labels:       <none>
Annotations:  <none>

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  app=nginx-example-efs

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Phase:  Completed

Backup Format Version:  1

Expiration:  2018-07-25 14:42:49 +0700 +07

Validation errors:  <none>

Persistent Volumes: <none included>

time="2018-06-25T07:42:50Z" level=info msg="PersistentVolume is not a supported volume type for snapshots, skipping." backup=heptio-ark/nginx-example-efs group=v1 groupResource=persistentvolumeclaims logSource="pkg/backup/item_backupper.go:307" name=pvc-b33e8de0-784a-11e8-957d-12dd8b001c9e namespace=nginx-example

[ncdc]

This is expected. EFS is an NFS-based file system, and there is no snapshot API available for it. Instead, you'll want to use our new integration with Restic. Note, there is a bug that we're currently working to address where you can't restore successfully.

[ncdc]

@rosskukulinski I don't know how we would necessarily do it, but it might be nice to find a way to somehow inform users that their volumes aren't getting backed up, or that they need to use Restic. I know we currently have a log message that you can find after the backup has completed, such as the one in the report above:

time="2018-06-25T07:42:50Z" level=info msg="PersistentVolume is not a supported volume type for snapshots, skipping." backup=heptio-ark/nginx-example-efs group=v1 groupResource=persistentvolumeclaims logSource="pkg/backup/item_backupper.go:307" name=pvc-b33e8de0-784a-11e8-957d-12dd8b001c9e namespace=nginx-example

but I wonder if we could make this more visible somehow.

[rosskukulinski]

@ncdc Good point. I would hope that #448 for backups would help solve this. In addition, we could tailor #550 to include a Pod Volume Backups.Ignored or Skipped.

[pmquang]

Hi @ncdc ,

I see restic example yaml used cloud-credentials, but I don't want to use it on production. I just used IAM role instead of. Is there any chance for it ?

[ncdc]

Yes, it works with IAM role credentials. If you have trouble getting it to work, please let us know.

…

On Tue, Jun 26, 2018 at 5:42 AM RAI ***@***.***> wrote: Hi, I see restic example yaml used cloud-credentials, but I don't want to use it on production. I just used IAM role instead of. Is there any chance for it ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#579 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAABYshyej-vrAgiE96j3GBw7vmP05Ggks5uAgIGgaJpZM4U1ush> .

[pmquang]

I mean in this yaml :

kind: DaemonSet
metadata: 
  name: restic
  namespace: heptio-ark
spec:
  selector:
    matchLabels:
      name: restic
  template:
    metadata:
      labels:
        name: restic
    spec:
      serviceAccountName: ark
      securityContext:
        runAsUser: 0
      volumes:
        - name: cloud-credentials
          secret:
            secretName: cloud-credentials
        - name: host-pods
          hostPath:
            path: /var/lib/kubelet/pods
        - name: scratch
          emptyDir: {}
      containers:
        - name: ark
          image: gcr.io/heptio-images/ark:latest
          command:
            - /ark
          args:
            - restic 
            - server
          volumeMounts:
            - name: cloud-credentials
              mountPath: /credentials
            - name: host-pods
              mountPath: /host_pods
              mountPropagation: HostToContainer
            - name: scratch
              mountPath: /scratch
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: HEPTIO_ARK_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: AWS_SHARED_CREDENTIALS_FILE
              value: /credentials/cloud
            - name: ARK_SCRATCH_DIR
              value: /scratch

I will remove all aws cloud-credentials and add annotation to use IAM role ( use kube2iam ). Will it work ?

[ncdc]

It should work

…

On Tue, Jun 26, 2018 at 7:21 AM RAI ***@***.***> wrote: I mean in this yaml : kind: DaemonSet metadata: name: restic namespace: heptio-ark spec: selector: matchLabels: name: restic template: metadata: labels: name: restic spec: serviceAccountName: ark securityContext: runAsUser: 0 volumes: - name: cloud-credentials secret: secretName: cloud-credentials - name: host-pods hostPath: path: /var/lib/kubelet/pods - name: scratch emptyDir: {} containers: - name: ark image: gcr.io/heptio-images/ark:latest command: - /ark args: - restic - server volumeMounts: - name: cloud-credentials mountPath: /credentials - name: host-pods mountPath: /host_pods mountPropagation: HostToContainer - name: scratch mountPath: /scratch env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: HEPTIO_ARK_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: AWS_SHARED_CREDENTIALS_FILE value: /credentials/cloud - name: ARK_SCRATCH_DIR value: /scratch I will remove all aws cloud-credentials and add annotation to use IAM role ( use kube2iam ). Will it work ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#579 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAABYqIruQC9TwNG9PslDRxyVaXbyaDtks5uAhlQgaJpZM4U1ush> .

[pmquang]

thanks @ncdc , let me try.

[pmquang]

hi @ncdc ,

I meet this error:

~ 123491$ kubectl logs --tail 100 restic-2jgld -n heptio-ark
Error: unknown command "restic" for "ark"
Run 'ark --help' for usage.
An error occurred: unknown command "restic" for "ark"

Could you help me check ?

[ncdc]

You need to change the tag for the image in the daemonset to v0.9.0.alpha.2 or master - whatever you used for the ark deployment.

…

On Wed, Jun 27, 2018 at 1:56 AM RAI ***@***.***> wrote: hi @ncdc <https://github.com/ncdc> , I meet this error: ~ 123491$ kubectl logs --tail 100 restic-2jgld -n heptio-ark Error: unknown command "restic" for "ark" Run 'ark --help' for usage. An error occurred: unknown command "restic" for "ark" Could you help me check ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#579 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAABYmZOGQdVl96KSrJWaHHe-N3iDDIEks5uAx6ogaJpZM4U1ush> .

[pmquang]

same @ncdc :)

~ 123491$ kubectl describe daemonset restic -n heptio-ark
Name:           restic
Selector:       name=restic
Node-Selector:  <none>
Labels:         name=restic
Annotations:    <none>
Desired Number of Nodes Scheduled: 29
Current Number of Nodes Scheduled: 29
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  29 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           name=restic
  Annotations:      iam.amazonaws.com/role=stg-heptio-ark-role
  Service Account:  ark
  Containers:
   ark:
    Image:  gcr.io/heptio-images/ark:v0.9.0.alpha.2
    Port:   <none>
    Command:
      /ark
    Args:
      restic
      server
    Limits:
      cpu:     300m
      memory:  300Mi
    Requests:
      cpu:     25m
      memory:  100Mi
    Environment:
      NODE_NAME:              (v1:spec.nodeName)
      HEPTIO_ARK_NAMESPACE:   (v1:metadata.namespace)
    Mounts:
      /plugins from plugins (rw)
  Volumes:
   plugins:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
Events:      <none>

~ 123491$ kubectl logs --tail 100 restic-pj98z -n heptio-ark
Error: unknown command "restic" for "ark"
Run 'ark --help' for usage.
An error occurred: unknown command "restic" for "ark"

[pmquang]

ah, it should be v0.9.0-alpha.2

[pmquang]

it still doesn't work @ncdc

[ncdc]

What errors do you get?

…

On Wed, Jun 27, 2018 at 6:13 AM RAI ***@***.***> wrote: it still doesn't work @ncdc <https://github.com/ncdc> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#579 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAABYkG9c7o818rgIjDDPI-EChBuxQF9ks5uA1q0gaJpZM4U1ush> .

[pmquang]

same @ncdc :

Error: unknown command "restic" for "ark"
Run 'ark --help' for usage.
An error occurred: unknown command "restic" for "ark"

And I don't know why I give wrong tag and k8s still can pull the image :|

[ncdc]

There are 29 instances of the restic pod (1 per node, and the output above shows that you have 29 nodes). It's possible you're looking at the logs from one of the older pods, before you set the image tag correctly. Please examine one of the new pods (created most recently) and confirm that its tag is correct, and then check to see if it's running / look at the logs.

[pmquang]

you're right :)

Its working now. Thank you.

[skriss]

Looks like this is resolved so closing out. Feel free to open a new issue if needed!