Vladhog Security: The great registry outage.
Harbor Registry
While preparing to release a new version of the Vladhog Security API, we discovered that we could no longer push images to our registry. In this post, we will explain what happened behind the scenes and why the outage occurred.
We were using Harbor registry. Despite the issues it has caused us in the past, we remained a fan for a while due to its strong access control, OAuth2 support, vulnerability scanning, and its ability to function as a proxy for other registries. Unfortunately, for reasons that were unclear at that time, Harbor suddenly began to break in ways we could no longer ignore.
A Harbor deployment consists of several key components:
Registry service, responsible for storing and returning Docker images, as well as performing garbage collection.
Core service, acting as both proxy server and API server, handling authorization requests and forwarding image pushes to the registry.
PostgreSQL database.
Redis cache.
Job service, responsible for tasks like garbage collection and vulnerability scanning.
When we push a Docker image, the request flows:
Load balancer → Harbor core → Harbor registry
Harbor core started resetting connections, even after changing timeout values and related settings. Redis and PostgreSQL were both stable, so the problem was clearly inside Harbor itself. We increased the number of Harbor core replicas, but those replicas also began losing connection to the registry. Eventually even registry replicas began failing.
After exhausting all options and unable to find a reliable fix, we decided that relying on Harbor was no longer viable. The resets continued, slowing our updates and deployment pipeline significantly. We began the search for alternative.
Searching for a Replacement
We needed only a few things from a new solution: Kubernetes-ready deployment, self-hosted, free and reliable.
Docker Registry, Github Registry, and Others
The official Docker Registry and Github Registry were not suitable for us. Docker allows only one private repository on the free tier, which is impossible for our workflow. Github Registry limits packages to 500 MB, which is barely fits a single Vladhog Security Bot build, let alone our API server.
Gitlab Registry (Cloud and Self-hosted)
Gitlab’s registry was a possible option. However, the free cloud tier offers only 10 GB of storage, which is nearly at our limit. The self-hosted version is solid and can be easily deployed with Helm, but we kept looking.
Nexus Repository
A popular solution, but with one major issue: Nexus does not provide licenses for self-hosted use inside the Russia Federation, which immediately ruled it our for us.
JFrog Artifactory
Also widely used, but entirely paid, even for self-hosted versions. Not an option.
Project Quay (Red Hat Quay, Quay.io)
A strong candidate, but Kubernetes deployment is complicated. It requires installing through OpenShift operator, bringing additional management components we did not need.
The Solution: Gitea (Self-Hosted)
A long time ago, we attempted a project called Cargo, where we used a customized Gitea instance for public use. Although that project didn’t succeed, we decided to revisit Gitea for our internal registry needs.
Gitea ended up having everything we required:
good documentation
solid access control
lightweight deployment
simple registry support
smooth Kubernetes setup
After deploying Gitea self-hosted, our registry problems disappeared. After nearly a week of Harbor troubleshooting and testing multiple alternatives, the solution turned out to be surprisingly lightweight and reliable.
Thank you for being with us!