johanneskueber.com

One TofuWorkspace per app: OpenTofu inside Crossplane, now with Keycloak clients

#crossplane #opentofu #terraform #keycloak #oidc #kubernetes #gitops #platform-engineering #selfhosted

Every app in my homelab needs a few things that do not live inside Kubernetes: a bucket for its backups, a DNS record so I can reach it over NetBird, and - since I retired LLDAP in October - an OIDC client in Keycloak. For a long time these were handled in a central OpenTofu environment on my laptop, plus a SOPS-encrypted secret per app into which I copied the client secret by hand. Adding an app meant touching three places, and removing one meant remembering all three. I usually remembered two.

What I wanted instead: the app folder in the GitOps repo describes everything the app needs, and the cluster takes care of the rest. Crossplane together with the OpenTofu provider gets me there - with one custom composite resource, the TofuWorkspace. Today 26 apps get their Keycloak client from it, and I have not opened the client list in the Keycloak admin console since.

The idea

Before building anything, I wrote down the following considerations:

  • One TofuWorkspace per namespace, a handful of lines of intent - no per-app Tofu module
  • Whatever already exists in the cluster is the source of truth: a k8up Schedule means “this app needs a backup bucket”, an internal HTTPRoute means “this app needs a DNS record”
  • The Terraform state is encrypted before it leaves the cluster
  • Secrets created by Tofu (the Keycloak client secret) land in the app’s namespace - nobody copies them anywhere
  • Access to an app is a Keycloak group membership, enforced by Keycloak itself
  • Deleting the app’s folder deletes everything the workspace created - cleanly, without a stuck namespace

The flow at the end looks like this:

flowchart LR
    git[(Git repo)] -->|Flux| xr[TofuWorkspace]
    sched[k8up Schedule] -.->|label opt-in| comp
    route[HTTPRoute] -.->|label opt-in| comp
    xr --> comp[Composition<br/>go-templating]
    comp --> pc[ProviderConfig<br/>S3 backend + encryption]
    comp --> ws[Workspace<br/>inline Tofu module]
    ws -->|tofu apply| ext["S3 buckets · NetBird DNS · Keycloak client"]
    ws -->|outputs| sec[Secret workspace-outputs]
    sec --> app[App / Envoy SecurityPolicy]

The XRD part

The Crossplane OpenTofu provider ships a Workspace resource that runs tofu against a module you give it. Used directly, every namespace would repeat the same backend block, the same provider declarations and the same “if this app also has backups, then…” logic. A Composite Resource Definition collapses all of that into one small, namespaced API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: tofuworkspaces.platform.example.com
spec:
  scope: Namespaced
  group: platform.example.com
  names:
    kind: TofuWorkspace
    plural: tofuworkspaces
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                s3Backend:
                  type: object
                  properties:
                    bucket:   { type: string }
                    key:      { type: string, default: "terraform.tfstate" }
                    endpoint: { type: string }
                  required: ["bucket", "endpoint"]
                retainBuckets:
                  type: boolean
                  default: true
                dns:
                  type: object
                  properties:
                    zoneId:  { type: string }
                    address: { type: string }
                    ttl:     { type: integer, default: 60 }
                  required: ["zoneId", "address"]
                oidc:
                  type: object
                  properties:
                    clientId:     { type: string }
                    name:         { type: string }
                    rootUrl:      { type: string }
                    redirectUris: { type: array, items: { type: string } }
                    webOrigins:   { type: array, items: { type: string } }
                    public:       { type: boolean, default: false }
                    pkce:         { type: string, enum: ["S256", "plain"] }
                    gated:        { type: boolean, default: true }
                    group:        { type: string }
                    render:
                      type: object
                      additionalProperties: { type: string }
                  required: ["name", "rootUrl", "redirectUris"]
              required: ["s3Backend"]

Some details:

  • scope: Namespaced: One workspace per namespace. The state, the secrets it reads and the secret it writes all share the namespace boundary - and with it the namespace’s RBAC.
  • s3Backend: The only required block. Everything else is optional and the Composition branches on its presence.
  • retainBuckets: Backup buckets are created with prevent_destroy. Flipping this to false (after emptying the bucket by hand) is the first step of decommissioning an app. More on that below.
  • oidc: The new part. The client id defaults to the namespace name, gated decides whether Keycloak itself refuses users without the right role, and group is the Keycloak group whose members get that role. I trimmed a few fields here (postLogoutRedirectUris, serviceAccount, optionalScopes) - they work the same way.

Nobody writes the full resource by hand, though. A Kustomize component carries the defaults (backend, bucket, DNS zone) and labels the namespace so the shared credentials get distributed into it. What an app adds is only its own part. This is the complete workspace definition for Gitea:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: platform.example.com/v1alpha1
kind: TofuWorkspace
metadata:
  name: workspace
spec:
  oidc:
    name: Gitea
    rootUrl: https://gitea.example.com
    redirectUris:
      - https://gitea.example.com/user/oauth2/Keycloak/callback
    group: developers

That is it. Gitea gets a Keycloak client called gitea, only members of developers can log in, and the secret shows up in the namespace.

The Composition part

The Composition runs in Pipeline mode with four steps:

  1. renderProviderConfig - function-go-templating renders the ProviderConfig with the S3 backend and the state encryption.
  2. renderWorkspace - the same function renders the Workspace with its inline Tofu module.
  3. renderUsages - protects the things the workspace needs for its own deletion.
  4. automatically-detect-ready-composed-resources - function-auto-ready derives the readiness of the XR.

The interesting trick is in step 2: the Tofu module is not static. It is templated from other Kubernetes objects in the namespace. The first version of this used function-extra-resources as a separate step for that, but since function-go-templating 0.13 a template can request those objects itself - and scoped to a namespace:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

---
apiVersion: meta.gotemplating.fn.crossplane.io/v1alpha1
kind: ExtraResources
requirements:
  k8upschedules:
    apiVersion: k8up.io/v1
    kind: Schedule
    namespace: {{ $xr.metadata.namespace }}
    matchLabels:
      crossplane-workspace: "true"
  httproutes:
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    namespace: {{ $xr.metadata.namespace }}
    matchLabels:
      crossplane-workspace: "true"

Some details:

  • namespace: Only objects in the XR’s own namespace arrive in the template. In the old setup I fetched everything cluster-wide and filtered in the template - which worked, but every workspace saw every route of every app.
  • matchLabels: The opt-in. Only objects carrying crossplane-workspace: "true" are visible to the Composition. The label is part of my backup and route components, so nobody has to remember it.
  • The first pass is empty: On the first run of a reconcile nothing has been fetched yet and .extraResources is empty. Crossplane reruns the pipeline once the requirements are known, so the template has to handle the empty case gracefully (| default dict everywhere).

Further down in the same template, the fetched objects become simple booleans and lists:

1
2
3
4
5
6

{{- $fetched := .extraResources | default dict -}}
{{- $k8upschedules := list -}}
{{- range ((index $fetched "k8upschedules") | default dict).items }}
  {{- $k8upschedules = append $k8upschedules .resource -}}
{{- end }}
{{- $hasBackup := gt (len $k8upschedules) 0 -}}

And {{- if $hasBackup }} blocks in the inline module decide which providers and resources are rendered.

The state part

The ProviderConfig carries the backend - and the part that makes me comfortable keeping the state in a bucket at Hetzner: OpenTofu’s native state encryption.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

terraform {
  backend "s3" {
    bucket = "tofu-workspaces"
    key    = "state.tfstate"
    region = "hel1"
    workspace_key_prefix = "gitea"

    use_path_style              = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    endpoints = { s3 = "https://hel1.your-objectstorage.com" }
  }

  encryption {
    key_provider "pbkdf2" "enc_key" {
      passphrase = var.state_encryption_passphrase
    }
    method "aes_gcm" "enc_method" {
      keys = key_provider.pbkdf2.enc_key
    }
    state { method = method.aes_gcm.enc_method }
    plan  { method = method.aes_gcm.enc_method }
  }
}

Some details:

  • workspace_key_prefix: One bucket for all workspaces, one prefix per namespace.
  • pbkdf2 + aes_gcm: The key is derived from a passphrase, state and plan are encrypted with it. Set both - the plan contains the same sensitive values as the state.
  • The passphrase: Comes from a SOPS-encrypted secret tofu-encryption in each namespace, passed in as TF_VAR_state_encryption_passphrase. Since the Keycloak client secret now lives in the state, this went from “nice to have” to mandatory.

Whoever reads the bucket sees an encrypted envelope and nothing else.

The backups and DNS part

This is what the workspace did before Keycloak came along, so only the short version:

  • Backups: For every labelled k8up Schedule, the module renders a bucket. My schedules go to Hetzner or Backblaze B2, so the template splits them by endpoint and renders the minio or the b2 provider. For B2 it additionally mints a per-bucket application key and exports it, so k8up never sees the account-wide key. Every bucket gets lifecycle { prevent_destroy = true } - a careless prune must never take the backups with it.
  • DNS: For every labelled HTTPRoute with an internal hostname, the module renders a netbird_dns_record pointing to the internal gateway VIP. An app getting an internal route is the trigger for its DNS record - no second declaration needed.

The Keycloak part

This is the part that changed the most in October. With spec.oidc set, the module gets a Keycloak provider and a client:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

provider "keycloak" {
  url           = "http://keycloak.keycloak.svc.cluster.local:8080"
  realm         = "homelab"
  client_id     = var.keycloak_client_id
  client_secret = var.keycloak_client_secret
}

resource "keycloak_openid_client" "oidc" {
  realm_id    = "homelab"
  client_id   = "{{ $client }}"
  name        = "{{ $oidc.name }}"
  description = "Managed by Crossplane: TofuWorkspace {{ $xr.metadata.namespace }}/{{ $xr.metadata.name }}. Edits in the admin console are overwritten."
  enabled     = true

  access_type = "{{ if $oidc.public }}PUBLIC{{ else }}CONFIDENTIAL{{ end }}"

  standard_flow_enabled        = true
  implicit_flow_enabled        = false
  direct_access_grants_enabled = false

  root_url            = "{{ $oidc.rootUrl }}"
  base_url            = "{{ $oidc.rootUrl }}"
  valid_redirect_uris = {{ $oidc.redirectUris | toJson }}
  web_origins         = {{ $oidc.webOrigins | default list | toJson }}

  authentication_flow_binding_overrides {
    browser_id = keycloak_authentication_flow.browser.id
  }
}

output "client-id" {
  value = keycloak_openid_client.oidc.client_id
}

output "client-secret" {
  value     = keycloak_openid_client.oidc.client_secret
  sensitive = true
}

Some details:

  • The provider URL: Tofu talks to the Keycloak Service inside the cluster. My admin hostname is a NetBird name the cluster cannot resolve, and the public route hides /admin - so the internal service it is.
  • The provider credentials: A dedicated service client crossplane with the realm-management roles, distributed into every workspace namespace by a ClusterExternalSecret. No admin password anywhere.
  • description: Sounds trivial, but it is my favourite line. Whoever opens the client in the admin console (usually me, months later) immediately sees that clicking around there is pointless.
  • No client_secret: Keycloak generates the secret, Tofu reads it back. I do not know the secret of any of my clients any more - and I do not need to.

Getting the secret to the app

The Workspace has writeConnectionSecretToRef set to workspace-outputs, so every Tofu output becomes a key in that secret. The keys are named client-id and client-secret on purpose: client-secret is exactly the key Envoy Gateway’s SecurityPolicy expects. For apps that sit behind the gateway login, the policy simply points at the secret:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: oidc-family
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: family
  oidc:
    provider:
      issuer: "https://keycloak.example.com/realms/homelab"
    clientID: "family"
    clientSecret:
      name: workspace-outputs
    redirectURL: "https://family.example.com/oauth2/callback"

Apps with their own OIDC login read it like any other secret, via secretKeyRef on workspace-outputs / client-secret. One name for every consumer.

And then there are the apps that want the secret embedded in something bigger. Paperless and Tandoor (both django-allauth) expect a JSON blob with the secret somewhere in the middle. For those, spec.oidc.render takes templates with a ${client_secret} placeholder:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

spec:
  oidc:
    name: Paperless
    rootUrl: https://paperless.example.com
    redirectUris:
      - https://paperless.example.com/accounts/oidc/keycloak/login/callback/
    group: family
    render:
      PAPERLESS_SOCIALACCOUNT_PROVIDERS: >-
        {"openid_connect":{"APPS":[{"provider_id":"keycloak","client_id":"paperless",
        "secret":"${client_secret}","settings":{"server_url":"https://keycloak.example.com/realms/homelab/.well-known/openid-configuration"}}]}}

Each entry becomes an additional output where the placeholder is replaced at apply time. The app reads the finished setting from workspace-outputs and has no idea Tofu was involved.

The gate

An OIDC client alone only answers who someone is. Everybody with an account in my realm could still log in to every app - and a lot of apps happily create a local account on the first OIDC login. The family should see Paperless; they have no business in Gitea.

Some apps can restrict access by group claims, most cannot. So the check moves into Keycloak itself: every gated client gets

  • a client role user,
  • its own copy of the realm’s passkey-first browser flow,
  • in that copy, right after the username form, a conditional Role Check subflow: if the user does not have <client>.user, deny access,
  • that flow bound as the client’s browser flow,
  • the role mapped to the group from spec.oidc.group.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

resource "keycloak_role" "user" {
  realm_id  = "homelab"
  client_id = keycloak_openid_client.oidc.id
  name      = "user"
}

resource "keycloak_authentication_flow" "browser" {
  realm_id  = "homelab"
  alias     = "gitea-browser-passkey-r4"
  copy_from = "browser-passkey"

  lifecycle {
    replace_triggered_by  = [terraform_data.flow_source]
    create_before_destroy = true
  }
}

resource "keycloak_authentication_subflow" "role_check" {
  realm_id          = "homelab"
  parent_flow_alias = "${keycloak_authentication_flow.browser.alias} passkey forms"
  alias             = "Role Check Gitea"
  requirement       = "CONDITIONAL"
  priority          = 15
}

resource "keycloak_authentication_execution" "role_condition" {
  realm_id          = "homelab"
  parent_flow_alias = keycloak_authentication_subflow.role_check.alias
  authenticator     = "conditional-user-role"
  requirement       = "REQUIRED"
  priority          = 10
}

resource "keycloak_authentication_execution_config" "role_condition" {
  realm_id     = "homelab"
  execution_id = keycloak_authentication_execution.role_condition.id
  alias        = "gitea.user"
  config = {
    condUserRole = "gitea.user"
    negate       = "true"
  }
}

resource "keycloak_authentication_execution" "deny" {
  realm_id          = "homelab"
  parent_flow_alias = keycloak_authentication_subflow.role_check.alias
  authenticator     = "deny-access-authenticator"
  requirement       = "REQUIRED"
  priority          = 20
  depends_on        = [keycloak_authentication_execution.role_condition]
}

resource "keycloak_group_roles" "users" {
  realm_id   = "homelab"
  group_id   = data.keycloak_group.users.id
  role_ids   = [keycloak_role.user.id]
  exhaustive = false
}

(In the Composition, gitea and Gitea are of course template variables - I resolved them here for readability.)

Some details:

  • copy_from: Duplicates the realm flow with all subflows, exactly like “Duplicate” in the admin console. Needs Keycloak provider 5.10 or newer.
  • The check sits after the username form: A user without the role is refused with “Access for user to client gitea not granted” before any credential is asked for. No passkey prompt for an app you are not allowed to use.
  • depends_on on the deny: Keycloak appends new executions in creation order. Without the dependency, the deny can end up in front of the condition - and then it denies everyone. Ask me how I know.
  • exhaustive = false: The group family holds the user role of many apps, each owned by its own workspace. With exhaustive = true every workspace would remove the roles of all the others on each apply.

Keeping 26 copies of a flow in sync

A copied flow does not follow later changes of its source. When I made the realm flow passkey-first in October, every client still had its old copy. The fix is a revision in the Composition:

1

{{- $flowSource := "browser-passkey@4" }}

terraform_data.flow_source takes this string as its input, and the flow has replace_triggered_by on it. Bumping the revision in a single line of the Composition replaces the flow copy (and the role check inside it) in every workspace on the next reconcile.

Note: Keycloak refuses to delete a flow that a client still has bound. A plain replace therefore fails - Tofu destroys first, Keycloak says no. create_before_destroy = true turns it around: create the new copy, rebind the client, then delete the old one. For that, both copies must exist side by side, which is why the revision is part of the alias (gitea-browser-passkey-r4). It took me three revisions to get this right; the -r4 is no coincidence.

The deletion part

Creating things was the easy part. Deleting an app was where it got ugly: Flux deletes the namespace together with the TofuWorkspace, and the namespace controller then removes every object in the namespace in parallel. That includes the ProviderConfig with the state backend and the secrets holding the credentials. The Workspace, meanwhile, needs exactly those to run tofu destroy - so it got stuck on “ProviderConfig not found”, and the namespace stayed in Terminating forever.

The third pipeline step renders a Crossplane Usage for every dependency:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: protection.crossplane.io/v1beta1
kind: Usage
metadata:
  name: gitea-workspace-uses-tofu-encryption
  namespace: gitea
spec:
  reason: "The Workspace reads credentials from this Secret while running tofu destroy"
  replayDeletion: true
  of:
    apiVersion: v1
    kind: Secret
    resourceRef:
      name: tofu-encryption
  by:
    apiVersion: opentofu.m.upbound.io/v1beta1
    kind: Workspace
    resourceRef:
      name: gitea

Some details:

  • Usage: Crossplane’s admission webhook rejects the deletion of the of object as long as the by object exists.
  • replayDeletion: Once the Workspace is gone, Crossplane replays the rejected deletion. Without it, the namespace controller would have to retry on its own schedule.
  • Which secrets: The step requests the same extra resources as the workspace step and renders a Usage only for the secrets that are actually in use - the Keycloak credentials only when spec.oidc is set, the backup credentials only when there is a schedule.

Combined with retainBuckets, decommissioning an app is now: empty the bucket by hand, set retainBuckets: false, wait for the reconcile, delete the folder. Keycloak client, flow, role, DNS record and bucket are gone afterwards.

The things I fine-tuned after some testing

  • Poll interval: The default reconciles every workspace every 10 minutes. Across all my namespaces that meant more than 200 tofu plan runs per hour against three external APIs - for buckets and DNS records that never drift on their own. Now the Workspace carries crossplane.io/poll-interval: "6h". Spec changes still reconcile immediately.
  • Pin the providers: Every Tofu provider in the inline module has an exact version, and tofu init runs without -upgrade. The pod resolves from its plugin cache instead of the registry on every run, and Renovate bumps the versions like everything else.
  • Sprig treats false as empty: {{ $xr.spec.retainBuckets | default true }} is always true, because default replaces false as well. Booleans with a default of true have to be resolved by hand with hasKey.
  • The label is silent: A route without crossplane-workspace: "true" simply gets no DNS record. No error, no event. Put the label into the component that creates the route.
  • Never import keycloak_group_roles: When I adopted my hand-made clients, the provider imported each group as exhaustive with all its roles. The next apply would have removed every role the workspace does not own. Let Tofu create the mapping instead - granting a role a group already has is a no-op.
  • Only activate what you use: The OpenTofu provider ships more managed resources than I need. A ManagedResourceActivationPolicy activates only workspaces.opentofu.m.upbound.io, so the API server watches one CRD instead of all of them.

Verifying

The XRs:

1
2
3
4
5

$ kubectl get tofuworkspace -A
NAMESPACE   NAME        SYNCED   READY   COMPOSITION
gitea       workspace   True     True    tofuworkspace
outline     workspace   True     True    tofuworkspace
...

The rendered module - the place to look when a provider block is missing:

1
2

kubectl get workspace.opentofu.m.upbound.io -n gitea gitea -o yaml \
  | yq '.spec.forProvider.module'

And what the app gets:

1
2
3

$ kubectl get secret -n gitea workspace-outputs -o json | jq -r '.data | keys[]'
client-id
client-secret

The real proof is a login: my own account gets into Gitea, the family account is stopped by Keycloak with “Access for user to client gitea not granted” - before it ever sees a passkey prompt.

Conclusion

The TofuWorkspace started as a small helper for backup buckets and has become the place where an app declares everything outside the cluster it depends on. The Keycloak part was the real test: 26 clients, each with its own gate and a secret that never touches my laptop or the Git repo. Adding a new app means writing ten lines of YAML; removing one means deleting its folder - and both actually work now.

The price is a 700-line Composition full of Go templates with HCL inside YAML. It is not pretty, and I would think twice before recommending it to a team. For a homelab with one maintainer, the trade-off is clearly worth it. If the template keeps growing, a typed language like KCL is the obvious next step - but that is a story for another post.

stat /posts/2026-10-31-crossplane-opentofu-workspaces/

2026-10-31: Initial publication of the article