Today in #docker on Freenode there was a person with a problem with their v1 Docker registry. I think I jinxed it when I said it was "extremely easy" to get a v2 registry running behind an Nginx proxy. It turned into a nightmare, and I'm sharing the design process to help anyone else that might need to debug problems with a similar setup.

So I had previously spent time getting a private registry to work behind the jwilder/nginx-proxy image, which is a great reverse proxy for docker containers. There is a lot of movement with Docker and especially the ecosystem of orchestration applications that exist around it. These days, plenty of stuff out there does what the nginx-proxy image does. Personally I like Consul and Serf from Hashicorp. Incidentally, they also make a hell of a nice application -- Vault -- that solves most of the problem with sharing sensitive configuration information. Anyways, for single-host-multiple-container environments, I still prefer proxying with the nginx-proxy image. I use it to front for all my web applications, and our Gitlab installation, so it only made sense to front my v2 registry with the same proxy.

The way the nginx-proxy image works is that it is bound to listen on tcp/80 and tcp/443 and uses the host machine's /var/run/docker.sock to listen for docker events, and then other containers are started with a VIRTUAL_HOST environment variable. When that happens, the docker-gen utility creates an Nginx template and starts routing requests to the container. You can also mount htpasswd files into the proxy and SSL certs to manage authentication and HTTPS connections. It also supports custom directives and custom templates.

So anyways, the problem the guy was having I was pretty sure he wouldn't be able to get support for, because v1 has been officially deprecated on Docker Hub, and also is no longer the primary registry endpoint the Docker client looks for when attempting to connect to a registry. Support for v2 was introduced in 1.6, and as of version 1.9, the client now prefers v2 registry endpoints over v1. So the time has come to upgrade.

Let's get started. First, pull the images we need to work with:

docker pull jwilder/nginx-proxy
docker pull registry:2.2

It's important to note that registry:latest does not point to the latest version of registry. The latest version points to v1! We need to make sure we're pulling v2. The docker registry is actually a part of the "distribution" repository. You can and should check for the latest version of both the v2 registry image and also any documentation there. It's generally true that the Docker documentation is mostly very good and correct, so make sure you read that and prefer that information above mine.

To configure the v2 registry, we need to create a minimal config.yml file. I usually keep all my stuff for docker under /var/docker/<container>, so sudo sh -c "mkdir -p /var/docker/registry && cd /var/docker/registry/ && vim config.yml" and put this into it:

version: 0.1
log:
    level: info
    formatter: json
    fields:
        service: registry
        environment: staging
        source: registry
http:
    addr: :5000
    host: myregistry.example.com
    secret: biglongsecretwhatever
storage:
    filesystem:
        rootdirectory: /var/lib/registry

This is the bare minimum you'll need to get your registry going. You should change http.secret to something long and random. For example you can use a bash one-liner like cat /dev/urandom | tr -dc a-zA-Z0-9 | head -c36 to get a random string. Now save this yaml file. Finally, mkdir /var/docker/registry/lib to create a directory for storing our registry images on the local host. I like to use AWS S3, and if this interests you, take a look at my last post on this subject for instructions.

We've got our v2 registry primed to run, but we won't run it quite yet.

Let's get the nginx-proxy image going. First mkdir /var/docker/nginx-proxy && cd /var/docker/nginx-proxy to get into our base host directory. Use mkdir vhost.d to create a directory for storing our special Nginx directives.

Now, if you want to have nginx-proxy handle your SSL certificates and authentication -- and we do, since docker will complain about a registry running without HTTPS -- you'll want to mdir htpasswd && mkdir certs.d at this time as well. For illustrative purposes, let's say our registry domain is myregistry.example.com and we've already pointed DNS at the host. So we'll name our SSL certificate to myregistry.example.com.crt and our key to myregistry.example.com.key and drop both of those files into the /var/docker/nginx-proxy/certs.d directory.

With HTTP authentication, we can just do some real basic stuff. We don't need anything fancy, since the client supports basic authentication. You can use htpasswd (from the apache2-utils repo on Linux Mint or Ubuntu) to generate authentication information. Save this information in /var/docker/nginx-proxy/htpasswd/myregistry.example.com similar to the way we named our SSL data. For future reference, you can also store a default certificate and key for HTTPS requests that arrive at your nginx-proxy which aren't routable to one of your containers.

We need a couple more files dropped into /var/docker/nginx-proxy/vhost.d. The first is myregistry.example.com and looks like this:

client_max_body_size 0;
chunked_transfer_encoding on;
 
location /v2/ {
  # Do not allow connections from docker 1.5 and earlier
  # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
  if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
    return 404;
  }
 
  add_header Docker-Distribution-Api-Version "registry/2.0";
  #more_set_headers     'Content-Type: application/json; charset=utf-8';
  include               vhost.d/docker-registry.conf;
}
 
location /v1/_ping {
  auth_basic off;
  include               vhost.d/docker-registry.conf;
  add_header X-Ping     "inside /v1/_ping";
  add_header X-Ping     "INSIDE /v1/_ping";
}
 
location /v1/users {
  auth_basic off;
  include               vhost.d/docker-registry.conf;
  add_header X-Users    "inside /v1/users";
  add_header X-Users    "INSIDE /v1/users";
}

These directives do a couple of things. First, they lift the artificial limit Nginx places on request size. Since you're going to be uploading huge files to your v2 registry, we need to turn off that limitation. This directive also prevents access by client versions 1.5 and below (which only use v1 registry endpoints anyway). Of particular interest is that the location directives for v1 endpoints have been proven to fix docker client bugs which caused the v2 registry to throw a 404 when trying to connect.

The second file is /var/docker/nginx-proxy/vhost.d/docker-registry.conf and looks like this:

proxy_pass                          http://myregistry.mexample.com;
proxy_set_header  Host              $http_host;   # required for docker client's sake
proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
proxy_set_header  X-Forwarded-Proto $scheme;
proxy_read_timeout                  900;

This forwards the IP address of your clients to the registry for logging purposes, as well as providing a couple of required headers.

Now that we've configured our v2 registry and nginx-proxy, let's start them up! We'll begin with the nginx-proxy container (shoutout to Arthur for noticing I'd forgotten to mount vhost.d in this next command):

docker run -d \
  --name "nginx-proxy" \
  --restart "always" \
  -p 80:80 \
  -p 443:443 \
  -v /var/docker/nginx-proxy/certs:/etc/nginx/certs:ro \
  -v /var/docker/nginx-proxy/htpasswd:/etc/nginx/htpasswd:ro \
  -v /var/docker/nginx-proxy/vhost.d:/etc/nginx/vhost.d \
  -v /var/run/docker.sock:/tmp/docker.sock \
  jwilder/nginx-proxy

Next, let's start our v2 registry container:

docker run -d \
  --name="registry" \
  --restart="always" \
  -v "/var/docker/registry/config/config.yml:/etc/docker/registry/config.yml" \
  -v "/var/docker/registry/lib:/var/lib/registry" \
  -e "VIRTUAL_HOST=myregistry.example.com" \
  registry:2.2

Notice here that we're not binding our v2 registry port to any port on the host. This is because we don't want to bind that port to the host interface. We want to "bind" that port to the nginx-proxy container. You may be wondering how the nginx-proxy container knows that it should route inbound HTTPS requests to port 5000 on the registry container. The answer is that it binds to whatever port your container has exposed, either via EXPOSE in the Dockerfile or --expose in the docker run command. In our example, the v2 registry image has EXPOSE 5000 in its Dockerfile.

This brings me to one last pain point I had with the nginx-proxy image. As wonderful as it is, it's not clear that you cannot proxy anything that isn't based on the HTTP protocol. So for example you can't have the nginx-proxy image front for your SMTP server or your FTP server. Also, something else that took me a while to understand... Let's say you want to run a cAdvisor container to expose some metrics for your Prometheus server. The documentation wants you to make that thing listen on tcp/8080. That's totally fine, but if you want to run this container behind your nginx-proxy container, you'll want to not bind that port to the host. And since it's exposed port 8080 in its Dockerfile, you can simply start it with -e "VIRTUAL_HOST=cadvisor.example.com" and it will then be available at https://cadvisor.example.com/, which will be served from behind your nginx-proxy. The nginx-proxy container gets an inbound HTTPS request on tcp/443, then routes the connection on the backend to tcp/8080 on your cAdvisor container. No --link necessary!

A piece of advice when debugging problems: Start with docker logs nginx-proxy and check if the problem is with your container or if it's with your nginx-proxy. Also you can use docker exec -it nginx-proxy /bin/bash to drop into a shell on your nginx-proxy container and poke around. You should check the /etc/nginx/conf.d/default.conf to make sure your container is being properly exposed to the nginx-proxy and its docker-gen utility.

Good luck!