Pattern for Dynamic Backup/Restore of Kubernetes Application Data

The Context

With container technologies like Docker and Kubernetes, Pods and Containers are ephemeral. All data is lost after a restart. The way to maintain application data after a restart is to attach a volume to the pod.
This doesn't address one thing however: the volume needs to reside on a storage backend and the volume will inherit the constraints of the storage backend.
Some of those constraints are:

High availability and resilience: The data may be deleted or be unavailable if the storage itself is removed or decommissioned. This is the case for local on-prem servers, for example.
Speed and price: some storage is either too slow or too pricey for some given use cases. NFS, for example, is a very common network storage but is not able to sustain high throughput scenarios.

A Possible Approach

The approach described here is to set up a dynamic backup and restore directly in the lifecycle of the Pod. The technology supporting this workflow is Restic.

Kubernetes backup workflow

Here is how the workflow works:

Init Container: The pod is associated with an Init container. On startup, the init container checks if the data is available locally or not. If the data is not available at the designated location, it will restore the snapshot from the backup storage (for example, a storage like S3).
Backup Container: A backup container will run in parallel and regularly create a new snapshot of data and send it to the backup storage.

Use Cases and Limitations

This pattern can be useful for some cases:

On-premise or home server: for applications running in an environment where the storage is not by nature fully resilient, this pattern gives an extra level of safety.
Dynamic infrastructure: for infrastructure that is designed to be completely decommissioned and re-created frequently, this can be a way to automate the restoration of application data. This is the case, for example, in a GitOps workflow where the infrastructure itself can be re-created.
Tests: having ephemeral storage that restores a version of a snapshot can be useful for a dynamic test environment, where the environment is dynamically started with the right test data.
Separation between runtime storage and backup storage: The runtime storage can be closer to the application whereas the backup storage can be cheaper and slower. This can be the case where you may want the application to store its data on a local block storage like AWS EBS, but the backup to be on a more resilient AWS S3.
There are some constraints and limitations that have to be taken into account for these kinds of workflows. If the local application data is completely ephemeral, or in case of loss of the local data infrastructure, the data loss will depend on the backup frequency. This kind of use may not be fully implemented depending on the update frequency of the data and on the criticality.

Implementation Example

The following tool is an example of implementation of this workflow: https://github.com/devopsplaybook-io/container-utils/

An example of a Kubernetes definition can be:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-application
  labels:
    app: my-application
spec:
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
    spec:
      containers:
        - image: my-application
          name: my-application
          volumeMounts:
            - mountPath: /data
              name: pod-volume
        - name: backup
          image: restic/restic:latest
          command: ["sh", "-c"]
          args:
            - wget -O /tmp/container-backup.sh https://raw.githubusercontent.com/devopsplaybook-io/container-utils/init/container-backup.sh && chmod +x /tmp/container-backup.sh && /tmp/container-backup.sh
          volumeMounts:
            - mountPath: /data
              name: pod-volume
          env:
            - name: BACKUP_FOLDER
              value: "/data"
            - name: BACKUP_RESTIC_REPO
              value: "... ..."
            - name: RESTIC_PASSWORD
              value: "... ..."
            - name: BACKUP_DO_PROCESS
              value: "Y"
            - name: BACKUP_DO_START_DELAY
              value: "10800"
            - name: BACKUP_DO_LOOP_FREQUENCY
              value: "10800"
      initContainers:
        - name: init
          image: restic/restic:latest
          command: ["sh", "-c"]
          args:
            - "wget -O /tmp/container-backup.sh https://raw.githubusercontent.com/devopsplaybook-io/container-utils/main/container-backup.sh && chmod +x /tmp/container-backup.sh && /tmp/container-backup.sh"
          volumeMounts:
            - mountPath: /data
              name: pod-volume
          env:
            - name: BACKUP_FOLDER
              value: "/data"
            - name: BACKUP_RESTIC_REPO
              value: "... ..."
            - name: RESTIC_PASSWORD
              value: "... ..."
            - name: BACKUP_DO_RESTORE
              value: "Y"
      volumes:
        - name: pod-volume
          emptyDir: {}