Tag: container registry

SSL Termination in front of Gitlab Runners

Setting up an internal url for a container registry on a self hosted Gitlab instance seemed like an easy task but I came across some unexpected results when pushing images. It worked fine before adding the SSL termination at the haproxy but once this was added I had some issues that were not well documented. I found lots of posts about this issue with lots of different suggestions, none of them seemed to work though.

Architecture Overview

Before diving into the problem, let me paint you a picture of my setup:

• k0s Kubernetes cluster hosting the GitLab Runner via Helm chart
• OpnSense firewall with the HAProxy plugin handling SSL termination
• GitLab instance sitting behind the firewall
• All traffic to both GitLab and the runners flows through OpnSense

The beauty of this setup is that SSL termination happens at the edge (OpnSense), simplifying certificate management. The challenge? Getting container registry operations to work reliably through this proxy layer.

The Symptoms: A Network Mystery

Everything seemed fine at first. The GitLab Runner was deployed, jobs were starting, but then came the errors during Docker image pushes. I saw lots of retries on the push to the docker registry and eventually it would start pushing the image layers but never get all the way thru the layers, sometimes it would stop in the middle of the layer. I would also get an error message such as the below.

write tcp 10.102.3.135:49990->10.6.0.1:5050: use of closed network connection

Docker-in-Docker issues?

My first suspect was the Docker-in-Docker (DinD) setup in the GitLab Runner chart. Running Docker inside Docker inside Kubernetes can introduce networking complexities, and I was convinced this was the culprit. I spent hours tweaking:

• DinD service configurations
• Gitlab configurations
• HaProxy configurations

But here’s where things got interesting…

The Plot Twist

I decided to test the registry connection from outside the k0s cluster. Using a simple docker login command worked perfectly which is why I didn’t think there was an issue outside of the pipeline but then found that when running docker push, the same timeout errors appeared!

This lead me to realize the problem wasn’t specific to the k0s cluster or the GitLab Runner setup, therefor was not a DinD issue at all. It was something more fundamental with how the registry was communicating through the HAProxy reverse proxy.

The Root Cause

After digging through GitLab’s registry documentation and multiple pages of people having the same issue with none of them working for me, I discovered the issue: GitLab’s container registry wasn’t properly handling the reverse proxy setup, particularly with how it constructed URLs for layer uploads. The registry was attempting to use relative URLs in a way that wasn’t compatible with the SSL termination happening at HAProxy which I thought was working correctly since docker login worked and some layers were correctly pushed before I received the error messages.

The Solution

The fix required two specific changes to the gitlab.rb configuration file:

Enable relative URL’s for the registry

# In gitlab.rb
registry['env'] = {
  'REGISTRY_HTTP_RELATIVEURLS' => "true"
}

This tells the Docker registry to use relative URLs when constructing responses, which works better behind a reverse proxy.

Specify the Full Registry Host URL

# In gitlab.rb
gitlab_rails['registry_host'] = "https://registry.yourdomain.com" 

adjust the registry host to match the url for your registry.

Notice the full URL including https:// – this is crucial! Many guides suggest using just the hostname, but when you’re behind a reverse proxy doing SSL termination, the registry needs to know the complete external URL to construct proper responses and after enabling relative URL’s you have to use the full URL here.

3. Apply the Configuration

After making these changes, save the file and reconfigure GitLab:

sudo gitlab-ctl reconfigure

The Results

After applying the fixes:

• ✅ No more “use of closed network connection” errors
• ✅ Image pushes completed significantly faster
• ✅ No more retries or partial uploads
• ✅ Consistent, reliable registry operations from both inside and outside the cluster

Lessons Learned

1. Test from multiple vantage points: Testing from outside the cluster helped me realize the problem wasn’t k0s-specific.
2. Don’t assume complexity: I immediately suspected the complex parts (DinD, Kubernetes networking) when the issue was actually in the simpler GitLab configuration.
3. Reverse proxies need special attention: When running services behind reverse proxies with SSL termination, always check if the service has specific configuration options for this scenario.
4. Full URLs matter: In proxy scenarios, services often need complete URLs (including the protocol) to function correctly, not just hostnames.
5. Docker login can work while docker push is still in a bad state.

Key Takeaways for Your Setup

If you’re running a similar setup with GitLab behind a reverse proxy:

1. Always enable REGISTRY_HTTP_RELATIVEURLS when using a reverse proxy
2. Use the complete external URL (including https://) for registry_host
3. Test registry operations from both inside and outside your cluster
4. Don’t immediately blame the most complex part of your stack
5. Error messages can be very cryptic sometimes and seem to point towards the issue lying somewhere that it is not.

Conclusion

What started as a frustrating network debugging session turned into a simple configuration fix. The combination of k0s, GitLab Runner, and OpnSense HAProxy works beautifully once you know these registry-specific settings.

If you’re seeing similar TCP connection errors or timeout issues with your GitLab registry behind a proxy, check these settings first. It might save you hours of debugging the wrong part of your stack.

Have you encountered similar issues with GitLab behind a reverse proxy? What other gotchas have you discovered? Let me know in the comments!