Selfhosted Continuous Integration - all parts required

As of late I came to wonder how dependent I want to be on cloud-solutions. After some researching I decided I wanted to be independent. This also includes leaving out great open source communities such as Github. There are plenty of great solutions for every aspect of the Software Development Lifecycle - most of them are hosted in the cloud, i.e. the computer of someone else. This is not only a problem once your internet is not working - but it also happens to become a problem when a certain service is shut down. Sometimes without warning. Sometimes forever. So i wondered: could I set up a basic continuous integration environment completely hosted in my environment.

The idea

Before starting my research, I wrote down the following considerations for building the setup:

• Everything is hosted on premise
• Everything is hosted as a Docker Container
• Smallest footprint possible (storage, CPU, RAM)
• Unified user management
• Everything works automatically but also manually if needed

The basic architecture of my self-hosted continuous integration service is as follows: we have a source code repository - Gitea, based on git - that is holding the source code for all the projects. This could be a homepage, a mobile app, or a full fledged server application. The source code repository sends webhooks to the build system - drone.ci - which builds container images based on the Dockerfiles in the repository. Upon successful completion of the build, images are pushed into the docker registry. The docker registry is scanned regularly for new images by watchtower, and if a newer image is available for a container, the image is updated and the container is restarted using the newly downloaded image.

Note: You might say that we already break the first consideration by using docker, but thats a risk I live with. Of course, docker is running on premise, the images are built locally, but the base images are and pulled from the cloud. Only our specific application layers are hosted inhouse. If docker decides to stop offering images we can no longer build new containers. However, what we already have stored will remain untouched and the service can be provided for as long as the image we have on premise is available. Therefor, I will accept this situation.

Gitea - Source code repository

Gitea (gitea.io) is used to store my git repositories storage solution. It looks hugely familiar as it is a clone (visually) of the github.com. However it can be installed locally in a small docker container and is really resource friendly. Based on go, the system requirements are low and the performance is great. Gitea is developed as an open source software and was forked some years ago from gogs.

Why not Gitlab (about.gitlab.com)? Well, Gitlab is very good and very widely adopted. It also features a lot of the stuff that I had to plug together manually. However, it is really resource hungry and hard to set up correctly. It easily takes a gig of RAM just for being there. It also uses programming languages I do not exactly like, i.e. Ruby. After initial tests I opted for Gitea and Drone, because it is a simpler setup, a smaller footprint, and not one monolithic application.

Drone - Build Server

Drone (drone.io) is used as the continuous integration build server. Drone is a go application that is able to react to repository changes and build new versions of an application. Using a configurable pipeline file (typically called .drone.yml) I am able to define - in code - how our artifacts should be built, tested, and published.

As mentioned above, Gitlab was a contender for this part of the toolchain as well. It features an integrated CI plugin, however it is hard to set up reliably (hello unicorn worker killer).

Docker Registry - Image repository

To store my built docker images I use the freely available Docker Registry:2 (docs.docker.com/registry). Other alternatives would have been the Sonatype Nexus Repository. It features a lot more than just a docker registry as it also serves as an npm or maven repository. However it uses a lot more resources and since all my builds are docker images I switched to the simple and lean Docker Registry. Also the docker registry is developed by the Docker team.

Watchtower - Automatic image deployment

In order to update the containers once a new version is available, I need to login and send the pull command to docker and bring the new container up. However, this is fairly tedious and boring. Instead we use a small service called Watchtower (https://github.com/containrrr/watchtower). Watchtower regularly checks for new versions of our images in the docker registry and if a new image is available, watchtower pulls and restarts the respective service. By giving each container a version number, we can limit watchtower to update only minor releases. Watchtower can also be disabled by default and activated for specific containers only.

Conclusion

Having set up all of the above mentioned tools has given me a self-hosted continuous integration/deployment toolchain. It can be tuned to my demands. I can automatically test commited code and build a new image if all tests are passed, I can automatically deploy the new version, and I can prepare images to be distributed to the outside (which would need additional user management).

Keep in mind that this setup is currently used for a single user only. I will further expand the system to allow fine-grained user management once I finished all parts of this tutorial. The user access will be based on Keycloak (keycloak.org)and OpenLDAP (openldap.org). The system is already in place, however I wanted to limit this series of blog posts to the basics to implement a CI environment on your own machine. In the future I will add to this topic.