Gitea Actions without Docker-in-Docker: every job a pod, every build on BuildKit

Back in 2020 I wrote about all the parts required for a self-hosted continuous integration: Gitea for the code, Drone for the builds, a registry for the images. Drone is long gone. Since December 2023 my builds have run on Gitea Actions, and the runner moved into the cluster on the same day as Gitea itself. Like almost every runner setup you find online, mine was a StatefulSet with two containers per pod: the runner, and a privileged docker:dind sidecar that actually ran the jobs.

It worked for almost three years. But it never really fit into Kubernetes. Recent versions of the Gitea runner ship a native Kubernetes backend, where every job becomes its own pod. So this week I switched. Container builds moved to a single, central BuildKit daemon, and the caches came along.

The old setup

For reference, this is what each of my four runner pods looked like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

containers:
  - name: gitea-runner
    image: docker.io/gitea/runner:5.2.4
    command:
      - sh
      - -c
      - |
        while ! nc -z localhost 2376; do
          echo 'waiting for docker daemon ...';
          sleep 5;
        done;
        exec /sbin/tini -- run.sh
    env:
      - name: DOCKER_HOST
        value: tcp://127.0.0.1:2376
  - name: dind
    image: docker.io/docker:29.9.3-dind
    args: ["--mtu=1000"]
    securityContext:
      privileged: true # needed for docker image builds

On top of that, the runner config had to smuggle the TLS certificates of the sidecar into every job container, so that a docker build inside a job could find the daemon:

1
2
3
4
5
6
7

container:
  options: "-v /certs:/certs:ro --add-host=docker:host-gateway"
  valid_volumes:
    - /certs
runner:
  envs:
    DOCKER_HOST: tcp://docker:2376

Why switch

Over time, the problems added up:

• A privileged container per runner. Every job ran next to a privileged daemon. One breakout and you are root on the node. And because of the sidecar, the whole gitea-runner namespace had to allow privileged pods.
• An image store that dies with the pod. dind keeps /var/lib/docker in its container layer. Every update of the runner or the dind image (and Renovate brings one almost every week) threw away all images. The first job on each runner then pulled the 2.3 GB ubuntu-latest job image again. On a busy node that took more than five minutes, during which the job log said docker pull and looked stuck.
• Kubernetes could not see the jobs. Requests, limits, priority classes and network policies applied to the runner pod as a whole. Whatever happened inside dind shared that budget, and the scheduler had no idea a heavy build was running. Meanwhile, each idle runner pod reserved 1100 MiB of memory just for being there.
• Its own network. A Docker bridge inside a pod inside a WireGuard overlay needs a smaller MTU; otherwise, large packets silently disappear. Hence --mtu=1000.

The idea

Before changing anything, I wrote down the following considerations:

• Image builds keep working with the usual docker/* actions, so the workflows barely change
• The shared actions cache stays as it is
• Every cache survives restarts and updates

The setup at the end looks like this:

flowchart LR
    gitea[Gitea] -->|jobs| runner[Runner StatefulSet<br/>2 replicas x 4 jobs]
    runner -->|create, exec, delete| job[Job pods]
    job -->|actions/cache| runner
    runner -->|relay| cache[(cache-server)]
    job -->|buildx remote driver| buildkit[BuildKit]
    buildkit -->|push| registry[(Gitea registry)]

Jobs as Kubernetes pods

With the Kubernetes backend, the runner no longer runs anything itself. It becomes a small controller: it polls Gitea for jobs, and for every job it creates a pod through the Kubernetes API. The pod gets a container named job from the job image, one more container for every entry in the workflow’s services:, and an emptyDir for the workspace. The runner then executes the steps inside the job container, streams the output back to Gitea and deletes the pod at the end. Every job pod carries an ownerReference to its runner pod, so if a runner disappears, Kubernetes cleans up its jobs as well.

The runner config

This is the complete config of my runner:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

runner:
  capacity: 4
  shutdown_timeout: 15m
  labels:
    - ubuntu-latest:kubernetes://docker.gitea.com/runner-images:ubuntu-latest@sha256:b96ba465021b1a4c6a1c9e79220c2b9bfad0f29b09dbb528bbfb35411443a3a4
  envs:
    BUILDKIT_HOST: tcp://buildkitd.buildkit.svc.cluster.local:1234

cache:
  enabled: true
  external_server: http://gitea-runner-cache.gitea-runner.svc.cluster.local:8088/
  external_secret_file: /etc/gitea-runner/cache/CACHE_SECRET

kubernetes:
  pod_template:
    metadata:
      labels:
        app.kubernetes.io/name: gitea-runner
        app.kubernetes.io/component: job
    spec:
      priorityClassName: tier-3
      imagePullSecrets:
        - name: regcred-docker-gitea
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: gitea-runner
              app.kubernetes.io/component: job
      containers:
        - name: job
          resources:
            requests:
              cpu: 250m
              memory: 1Gi
            limits:
              memory: 6Gi
        - name: $services
          resources:
            requests:
              cpu: 50m
              memory: 256Mi
            limits:
              memory: 2Gi

Some details:

• capacity: 4: Jobs per runner. Before, each runner ran one job at a time next to its own daemon. Now it is just a budget: two replicas with four slots each run eight jobs in parallel.
• shutdown_timeout: 15m: A stopping runner (update, node drain) takes no new jobs and gives the running ones 15 minutes. The pod’s terminationGracePeriodSeconds has to sit above that (I use 960 seconds); otherwise, Kubernetes kills the runner first - and through the ownerReference all of its jobs with it.
• labels: The part after the first colon decides which backend runs a job: docker:// would start it on a Docker daemon, and kubernetes:// runs it as a pod with that image. The workflows do not notice: runs-on: ubuntu-latest stays runs-on: ubuntu-latest. The label overrides whatever the runner stored when it registered and is sent to Gitea on every start, so a new job image is just a config change. The image is pinned by digest. With a plain tag, every node would keep whatever ubuntu-latest was when it first pulled it.
• envs: Environment variables for every job. BUILDKIT_HOST tells the jobs where to build images - more on that below.
• kubernetes.pod_template: Merged into every job pod. The job container and every service container ($services stands for all of them) get requests and limits. The labels are what the network policies select on. tier-3 is my lowest priority class and never preempts anything, so a full cluster delays CI instead of evicting applications. The topology spread keeps eight parallel jobs from piling up on one node.

Permissions

The runner needs a service account that may create pods, exec into them, read their logs and delete them. It also creates a Secret per job pod, which carries the environment of the service containers. Everything stays inside the runner’s namespace:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gitea-runner
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "get", "list", "delete"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "delete"]

Secrets are create and delete only. The runner cannot read or list the other Secrets in the namespace.

The StatefulSet itself shrinks to a single container with the stock entrypoint, the service account and the 1 GiB volume that keeps its registration. The runner requests 10m CPU and 128 MiB now, since it only schedules and relays.

The namespace now enforces the Pod Security baseline level instead of privileged. The runner is allowed to create pods in this namespace, so this level is what stops a compromised runner from starting a privileged one. restricted would be even better, but the runner images run as root, so it stays at warn and audit for now.

What no longer works

There is no Docker daemon in a job, and the runner fails a job right away if it needs one:

• Docker actions (runs.using: docker in the action.yml)
• uses: docker://... steps
• container.options, container.volumes and container.credentials
• Anything that calls docker run or docker build --load

To find every affected workflow, I did not grep my local checkouts. Gitea’s database knows every workflow that ever ran (table action_run), including those in repositories I have not cloned in years. Most workflows did not care at all. The ones that did:

• Private job images. One pipeline runs inside an image from my Gitea registry and used to pull it with container.credentials. Now the kubelet pulls the image, so the credentials moved into the imagePullSecrets of the pod template, and the workflow lost three lines.
• Image builds. These moved to BuildKit (next section).

Last detail for this part: Renovate does not understand the kubernetes:// label out of the box. A small regex manager keeps the job image and its digest up to date:

1
2
3
4
5
6
7
8

{
  "customType": "regex",
  "managerFilePatterns": ["/^k8s/apps/gitea-runner/.+\\.ya?ml$/"],
  "matchStrings": [
    "kubernetes://(?<depName>[^:@\\s\"']+):(?<currentValue>[^@\\s\"']+)@(?<currentDigest>sha256:[a-f0-9]{64})"
  ],
  "datasourceTemplate": "docker"
}

BuildKit - one daemon for all builds

Without a daemon, a job cannot build container images on its own. I looked at the following options:

• A BuildKit sidecar in every job pod - still privileged, now in the namespace where the runner creates pods, and every build starts with an empty cache.
• Kaniko or Buildah inside the job - unprivileged, but different tools with their own quirks, no buildx, and every workflow would need a rewrite.
• Rootless BuildKit - would be my favourite, but it needs unprivileged user namespaces. Talos disables them (user.max_user_namespaces = 0), and I do not want to change that just for CI.

I went with one central BuildKit daemon. BuildKit is the build engine inside docker build anyway, and it also runs standalone as buildkitd, reachable over gRPC. buildx can drive such a daemon with its remote driver as if it were a local one. The one component that really needs privileges sits alone in its own namespace, all builds share one warm cache, and every workflow changes by two lines.

The daemon

BuildKit is configured with a single buildkitd.toml, mounted from a ConfigMap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

[grpc]
  address = ["unix:///run/buildkit/buildkitd.sock", "tcp://0.0.0.0:1234"]

[cdi]
  disabled = true

[worker.oci]
  enabled = true
  networkMode = "host"
  gc = true
  reservedSpace = "20%"
  maxUsedSpace = "80%"
  minFreeSpace = "10%"
  max-parallelism = 4

[worker.containerd]
  enabled = false

Some details:

• address: The Unix socket is for the probes inside the pod (buildctl uses it by default), and TCP port 1234 is for the job pods.
• networkMode = “host”: Sounds scary, but “host” means the network namespace of the BuildKit pod, not the node. RUN steps share it, so they get the pod’s MTU and fall under the pod’s network policies. A bridge would bring back the MTU hack.
• gc: The limits are percentages of the cache volume. If I grow the volume, the cache grows with it, and the config stays the same.
• max-parallelism = 4: Build steps running at once across all clients. Eight parallel jobs queue for the CPUs instead of fighting over them.

The Deployment around it is short:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: apps/v1
kind: Deployment
metadata:
  name: buildkitd
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    spec:
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 900
      containers:
        - name: buildkitd
          image: docker.io/moby/buildkit:v0.34.2
          args:
            - --config=/etc/buildkit/buildkitd.toml
          ports:
            - containerPort: 1234
              name: grpc
          readinessProbe:
            exec:
              command: ["buildctl", "debug", "workers"]
          securityContext:
            privileged: true
          # volumes: the ConfigMap at /etc/buildkit, the cache PVC at /var/lib/buildkit

Some details:

• privileged: true: Rootful BuildKit creates namespaces and mounts for every build step, and that needs privileges. The buildkit namespace allows privileged pods and contains nothing else.
• Recreate: buildkitd locks its state directory, and the volume is ReadWriteOnce. Two daemons must never overlap.
• terminationGracePeriodSeconds: 900: buildkitd stops gracefully on SIGTERM. Running builds get 15 minutes to finish before the new version takes over the cache.

A Service named buildkitd exposes port 1234, and that is the BUILDKIT_HOST from the runner config.

Note: The gRPC port has no authentication and no TLS. Whoever reaches port 1234 can run arbitrary build steps on a privileged daemon. That makes the network policy the access control, and it only lets in job pods from the runner namespace:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ingress-buildkitd
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: buildkit
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: gitea-runner
          podSelector:
            matchLabels:
              app.kubernetes.io/name: gitea-runner
              app.kubernetes.io/component: job
      ports:
        - protocol: TCP
          port: 1234

These are the labels from the pod template above. The runner pods themselves cannot reach BuildKit; only the jobs can.

The workflow change

In every workflow that builds an image, the buildx setup gets two more lines:

1
2
3
4
5

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4
        with:
          driver: remote
          endpoint: ${{ env.BUILDKIT_HOST }}

BUILDKIT_HOST comes from the runner’s envs and is available in the env context. Everything else stays: docker/login-action works without a daemon (the login happens in the CLI, and buildx hands the credentials to BuildKit for the push), and docker/build-push-action builds and pushes as before.

What is gone is the local image store. load: true does not work, and neither does a docker push after the build - BuildKit has to push the image itself. For signing, take the digest from the build step’s output. cosign reads from the registry anyway.

Caching

There are three caches now, and all of them survive restarts.

The actions cache

actions/cache, setup-node with cache: npm and friends store their data in the actions cache. I already ran a shared cache server for all runners - the runner binary’s own cache-server subcommand, as a separate Deployment on a 20 GiB Longhorn volume - and it stays as it is. Jobs do not talk to it directly. Their ACTIONS_CACHE_URL points to a small forwarding listener in their own runner, which relays to the cache server. Job pods and runners share a namespace, so that just works.

The only change: the shared secret between runner and cache server no longer needs an init container with sed. Newer runners read it from a file (external_secret_file in the config above), mounted from a Kubernetes Secret.

The job image

The job image is now pulled by the kubelet, like every other image in the cluster. It lands in containerd’s image store on the node and stays there across runner updates, job pods and restarts. A node pulls the 2.3 GB once per image version, not once per runner rollout. Since the label pins the digest, all nodes run exactly the same image.

On top of that, I run Spegel, a peer-to-peer registry mirror on every node. When a node pulls an image a neighbour already has, containerd fetches the layers from the neighbour instead of the internet. On the day of the switch, about a third of the requests for the new job image were already answered by another node.

The BuildKit layer cache

BuildKit caches every build step by its inputs. With dind, this cache lived in each runner’s daemon and vanished with every restart. Now there is exactly one cache for all builds of all repositories, and it lives on its own 50 GiB volume. The GC rules from the buildkitd.toml keep usage below 80% of that.

The volume uses longhorn-object-single-encrypted, my storage class for disposable data. It is encrypted (see per-PVC encryption with Longhorn), but has a single replica only. Losing the volume costs one cold build per image, so nothing in there is worth replicating three times.

Builds that need their cache somewhere else can still export it to the registry with cache-to: type=registry. The local cache is used either way.

Conclusion

The switch itself was quick. Most of the time went into finding every workflow that used something the new backend does not support. The result:

• Eight jobs in parallel instead of four, each with its own requests and limits, and visible to the scheduler
• The runner pods went from 1100 MiB memory requests each to 128 MiB

Keep in mind that there are downsides. BuildKit is a single daemon: if it is down, no image gets built. For a homelab I can live with that. The cache volume has a single replica, so if the node holding it dies, BuildKit waits until the node is back - or until I delete the volume and accept a round of cold builds. And workflows built around a Docker daemon need rewriting. docker compose up for integration tests does not work anymore - services: in the job pod take its place.

The runner also offers an S3 backend for the actions cache, which could replace the cache server with a bucket. That might be the next step.