A Backup Target on Backblaze B2 - k8up, Crossplane, and External Secrets in One Loop

All my restic backups used to go to exactly one place: Hetzner Object Storage. That is a fine warm tier, except for one detail - the cluster’s infrastructure also runs on Hetzner. A provider outage, a billing mishap, or a compromised account takes out the workloads and their backups in the same event. I had off-site backups on Scaleway once, but their object storage made restic crawl badly enough that I removed it and never looked back.

So: another backup location, explicitly not Hetzner, and cheap enough that “back up everything twice” doesn’t need a budget discussion. I landed on Backblaze B2 (EU region). The price is right, restic over its S3 API is a well-worn path, and - the part that decided it - B2 has a first-party Terraform provider that manages not just buckets but application keys. That last bit turns a chore into a pattern: every app gets its own bucket and its own credential scoped to exactly that bucket, minted by machinery instead of by me.

This post is about the machinery. Three operators that usually mind their own business - k8up, Crossplane, and External Secrets - end up passing work to each other in a loop, and the result is that adding a backup target to ~34 applications was one new file in a shared kustomize component.

The loop

flowchart LR
    sched["k8up Schedule<br/>(labeled, per app)"] -->|watched via<br/>extra-resources| comp["Crossplane<br/>TofuWorkspace composition"]
    eso["External Secrets<br/>ClusterExternalSecret"] -->|account key,<br/>workspace namespaces only| comp
    comp -->|b2_bucket +<br/>b2_application_key| b2[("Backblaze B2")]
    comp -->|"outputs →<br/>connection secret"| out["workspace-outputs<br/>Secret"]
    out -->|"secretKeyRef"| sched2["k8up backup job<br/>(restic)"]
    sched2 -->|encrypted snapshots| b2

Read it as a chain of intent: the k8up Schedule declares that a namespace wants backups on B2. Crossplane observes that declaration and provisions what is missing - the bucket and a per-bucket key. The key comes back as a Kubernetes Secret, and k8up consumes it without ever knowing how it got there. External Secrets feeds the one credential a human actually created, and only to the namespaces that run provisioning.

Nobody creates buckets by hand. Nobody copies per-app credentials around. There is exactly one secret in the whole design that I typed in, and no application ever sees it.

One Schedule, 34 applications

The backups already follow a kustomize component pattern: a shared base holds a single Schedule, per-environment and per-app components stack transformers on top that rewrite the bucket name (workload → workload-talos-gitea-<id> and so on). The B2 target is a sibling of the existing Hetzner Schedule in that shared base:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: schedule-backblaze
  labels:
    crossplane-workspace: "true"
spec:
  backend:
    envFrom:
      - secretRef:
          name: k8up-backblaze-config   # AWS_DEFAULT_REGION
    s3:
      endpoint: s3.eu-central-003.backblazeb2.com
      bucket: workload
      accessKeyIDSecretRef:
        name: workspace-outputs
        key: backup_key_schedule_backblaze_id
      secretAccessKeySecretRef:
        name: workspace-outputs
        key: backup_key_schedule_backblaze_secret
    repoPasswordSecretRef:
      name: k8up-restic-password
      key: password
  backup:
    schedule: "@daily-random"
  check:
    schedule: "@weekly-random"
  prune:
    schedule: "@weekly-random"
    retention: { keepLast: 5, keepDaily: 7, keepWeekly: 4, keepMonthly: 12, keepYearly: 5 }

Because the bucket-name transformers match on kind: Schedule rather than on a resource name, the new Schedule inherits the per-app naming for free. Every app that includes the backups component now has two Schedules - same PVCs, same restic password, same pre-backup pg_dump annotations, two independent repositories on two providers. k8up’s @daily-random gives each Schedule its own stable slot, so the two runs don’t trample each other.

The interesting part is the two secretKeyRefs: they point at a Secret called workspace-outputs that does not exist in Git. It is the connection secret of a Crossplane-managed OpenTofu workspace, and its contents are minted per namespace.

The composition: intent in, credentials out

I already run OpenTofu inside Crossplane through a small TofuWorkspace XR - one per namespace, with the inline module templated from other Kubernetes resources in that namespace (I wrote that pattern up in an earlier post). The composition pulls in every k8up Schedule labeled crossplane-workspace: "true" via function-extra-resources and routes each one by its endpoint:

1
2
3
4
5
6
7

{{- range $schedule := $nsK8upSchedules }}
  {{- if contains "backblazeb2.com" $schedule.spec.backend.s3.endpoint }}
    {{- $backblazeSchedules = append $backblazeSchedules $schedule -}}
  {{- else }}
    {{- $hetznerSchedules = append $hetznerSchedules $schedule -}}
  {{- end }}
{{- end }}

The Schedule is the single source of truth: its bucket field (already rewritten per app by kustomize) names the bucket to create, its endpoint decides which provider block handles it. For each B2 schedule, the rendered module is pure first-party provider - no generic S3 shim:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

resource "b2_bucket" "backup_bucket_schedule_backblaze" {
  bucket_name = "workload-talos-gitea-180e90fe"
  bucket_type = "allPrivate"
  lifecycle { prevent_destroy = true }
}

resource "b2_application_key" "backup_key_schedule_backblaze" {
  key_name     = "workload-talos-gitea-180e90fe"
  bucket_ids   = [b2_bucket.backup_bucket_schedule_backblaze.bucket_id]
  capabilities = ["listBuckets", "listFiles", "readFiles", "writeFiles", "deleteFiles"]
}

output "backup_key_schedule_backblaze_id" {
  value = b2_application_key.backup_key_schedule_backblaze.application_key_id
}

output "backup_key_schedule_backblaze_secret" {
  value     = b2_application_key.backup_key_schedule_backblaze.application_key
  sensitive = true
}

The outputs are the handover. The OpenTofu provider writes every module output into the workspace’s connection secret - workspace-outputs, in the same namespace - and that is precisely the Secret the k8up Schedule references. Marking the key sensitive keeps it out of the workspace’s status; it exists only inside the Secret.

That closes the loop: Schedule → composition → B2 → connection secret → Schedule. The credential k8up runs with was never in Git, never in SOPS, never on my clipboard.

One human secret, fenced in by External Secrets

The composition needs something to talk to B2 with: an account-level application key that may create buckets and mint keys. That is the one secret I created, and External Secrets controls exactly where it lands. The hub copy is SOPS-encrypted in a secrets-hub namespace; a ClusterExternalSecret fans it out - but only to namespaces labeled for workspace provisioning, not to every namespace with backups:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: external-secrets.io/v1
kind: ClusterExternalSecret
metadata:
  name: backblaze-account-creds
spec:
  namespaceSelector:
    matchLabels:
      secrets.tofu-workspace: "true"
  externalSecretSpec:
    secretStoreRef: { kind: ClusterSecretStore, name: secrets-hub }
    target: { name: backblaze-account-credentials, creationPolicy: Owner }
    dataFrom:
      - extract: { key: backblaze-account-credentials }

The workspace consumes it as TF_VARs; k8up never references it. And the key itself is deliberately crippled - it carries only management capabilities:

1
2

b2 key create workspace-provisioner \
  listBuckets,readBuckets,writeBuckets,listKeys,writeKeys,deleteKeys

No readFiles, no writeFiles, no deleteFiles: the provisioning key cannot touch backup data. The per-bucket keys it mints can - but only inside their own bucket, and bucket-restricted keys can’t hold writeKeys, so a leaked app credential cannot escalate. The honest caveat: anything holding writeKeys can mint itself a stronger key, so the account key is still root-in-waiting. That is exactly why it stays fenced to the workspace namespaces instead of riding along with every k8up credential.

Bootstrapping is just eventual consistency

There is an obvious ordering problem: the Schedule references a Secret that only exists after the workspace has run, and the workspace only runs after Flux applies everything. I did nothing about it, on purpose. The first k8up job in a namespace fails with a missing secret, Crossplane reconciles the workspace, the bucket and key appear, workspace-outputs materializes, and the next scheduled run succeeds. Every controller involved retries; “the credential doesn’t exist yet” is just Tuesday in a Kubernetes cluster. Resist the urge to orchestrate it.

Pitfalls that cost me time

• Web-console keys can’t provision. Keys created in the B2 browser UI never get writeKeys/deleteKeys - those capabilities are CLI/API-only. If the workspace fails with unauthorized while minting keys, this is why.
• The account’s region is the bucket’s region. b2_bucket has no region argument; buckets land wherever the account was created. The Schedule endpoint (s3.eu-central-003...) and AWS_DEFAULT_REGION must match the account, not the other way around.
• listBuckets on restricted keys. A bucket-restricted key without the listBuckets capability breaks most S3 SDKs (and restic) on the compatibility API. It’s in the capability list above for a reason.
• ClusterSecretStore RBAC. My secrets-hub store’s Role whitelists readable Secrets via resourceNames. A new hub secret silently fails to sync until it’s added to that list - the ExternalSecret just sits in SecretSyncedError.
• Import by ID, not name. Adopting a pre-existing B2 bucket into state means import with the bucket ID. One more way B2’s native provider differs from S3 muscle memory.

Why I like this shape

Each operator does the one thing it is good at: kustomize stamps out per-app intent, External Secrets moves the single human credential to exactly the blast radius it belongs in, Crossplane turns declared intent into external resources, and k8up just runs restic with whatever credentials appear. Adding a third provider some day is a template branch and a provider block - about forty lines - and every app in the cluster picks it up on the next reconcile.

The full wiring, with repo paths and verification commands, lives in the homelab docs.