Recently, I was offering some help on Stack Overflow to someone asking about deleting images from a private Docker registry. Here I mean a v2 registry, which is part of the Docker Distribution project. I should advise anyone reading this that no one ever refers to a v1 registry anymore: That project is dead, even though it occupies the registry:latest image tag on Dockerhub (you want to pull registry:2 at least). The v1 registry is an old Python project, and v2 is written in Go. Anyways, the v2 registry has not had delete capabilities (via the API) since its inception. This was my initial assumption, but I took the opportunity to research the latest information.

As it turns out, the latest versions of registry (later than v2.4 I think) do have delete functionality. While the main Docker project ("docker-engine") has excellent documentation, the Distribution project has previously not been. It's not bad, but it's not great, either. The API documentation is not very clear on using the new delete API functionality. But it's there, along with an interesting garbage collection mechanism. That's a topic for another day, but it's the reason I wanted to upgrade to version 2.4 of the registry. I was using version 2.1 or something.

Cue an upgrade, and a couple of problems. First, in the config.yml file, in the cache section under the storage section, the layerinfo setting has been deprecated and renamed to blobdescriptor. That isn't a blocking change yet, but it will be soon, so rename it now while you have the chance.

Finally, if you're backing your registry with S3 like a sane human being, the permissions have changed and the change is not documented anywhere. Zing! I couldn't push when I fired up my new registry container. I kept getting "Retrying in X seconds" messages when pushing individual layers. I killed and deleted the container, started up a new one with the level setting under the log section set to debug in my config.yml file. This yielded the key to the issue (notice the "s3aws: AccessDenied" message and 403 status code):

"err.code":"unknown","err.detail":"s3aws: AccessDenied: Access Denied\n\tstatus code: 403, request id: 11E0123C033B0DB5","err.message":"unknown error"

Here's what the new S3 policy needs to look like (beware copying from the documentation linked above: There is an errant comma in the documented policy):

 "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:ListBucket",
          "s3:GetBucketLocation",
          "s3:ListBucketMultipartUploads"
        ],
        "Resource": "arn:aws:s3:::mybucket"
      },
      {
        "Effect": "Allow",
        "Action": [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
          "s3:ListMultipartUploadParts",
          "s3:AbortMultipartUpload"
        ],
        "Resource": "arn:aws:s3:::mybucket/*"
      }
]

Just for the record, this new policy adds support for the s3.GetBucketLocation, s3:ListBucketMultipartUploads Actions on your particular bucket, and the s3:ListMultipartUploadParts and s3:AbortMultipartUpload Actions on your bucket contents.