Kubernetes is like other container orchestration platforms, except that it has many layers of abstraction built into it with the idea that these abstractions allow you more headroom to scale. The thing is that these abstractions make Kubernetes far, far more complex than the vast majority of organizations need to leverage. Bear with me for a minute... Kubernetes was built by Google to orchestrate its services. But your company is not Google and there are many more important problems on the road between your company and Google. You do what whatever you want, but I don't spend time solving problems I don't have. And we haven't even touched on the problems you're creating by introducing so many layers of abstraction into your environment. At some point, many of the folks using Kubernetes will hit its complexity like a brick wall. They may not be able to extricate themselves because of sunk cost. Fortunately for them, Docker Swarm is in a position to serve as a lifeline for these people to automate away much of the complexity around Kubernetes and bring their environment and orchestration back under their control again.

This is more or less just a post for myself. I always end up dumping a couple of hours into this problem whenever I get a new machine for work -- surprise! I work for Docker now -- and tonight especially I really could have used this post instead of wasting that time researching the problem all over again. I choose Thinkpads when I have a choice, because the popular alternative is stupid.

Anyway, I use Linux Mint with Cinnamon, and LM18 is the current version. It's based on Ubuntu 16.04. I've chosen a P50 and upgraded the RAM to 64GB. Everything works out of the box, including the weird dual graphics situation going on under the hood. However, I want a super sensitive Trackpoint. The sensitivity settings are under something like /sys/devices/platform/i8042/serio1/serio2/ in the sensitivity, speed and inertia files. I like to keep mine at about 255, 230, and 4, respectively. The value of 255 is maximal, btw.

Now, simply dumping my preferences into those files works for the current session. Meaning, when I reboot the machine, they are reset to their defaults. So I'm using systemd to write values into these files on boot. I've got a /etc/tmpfiles.d/tpoint.conf file with the following contents:

w /sys/devices/platform/i8042/serio1/serio2/speed - - - - 230
w /sys/devices/platform/i8042/serio1/serio2/sensitivity - - - - 255
w /sys/devices/platform/i8042/serio1/serio2/inertia - - - - 4

Now, this works great. However, after I resume (or thaw) from a suspend (or hibernate), the location of these config files changes. So what I've done is added a simple shell script to /etc/pm/sleep.d/trackpoint-fix which contains the following:

#!/bin/bash
# set sensitivity/speed of trackpoint on resume
case "${1}" in
	suspend|hibernate)
		# suspending to RAM
		sleep 0
		;;
	resume|thaw)
		# resume from suspend 
		newdir=$(find /sys/devices/platform/ | grep sensitivity | sed -e "s/sensitivity//")
		echo 230 | sudo tee > ${newdir}speed
		echo 255 | sudo tee > ${newdir}sensitivity
		echo 4 | sudo tee > ${newdir}inertia
		;;       
esac   

This is pretty self-explanatory. I dig around for the new location of these configuration files and then dump my favorite values into them. That's all there is to it.

Hey! You might have seen my recent post about having Dockerized some software called GraphLab Create (together with IPython) for a machine learning course I was taking. As it happens, I've found that image so useful for other, generic ML work that I've pared it down to its IPython/Anaconda bundle only. So I'd like to introduce the super-simple but super-useful Dockerized IPython / Anaconda project!

This repo includes a couple of useful scripts: One for building the image (build.sh) and one for running the resultant image as a container (run.sh). Just run the build script and then the run script (and optionally provide a directory to mount into the container for data files) and you're all set! Note: Either your specified directory or current working directory will be mounted to /data as a volume into the container! Also, your IPython Notebook may include import statements which reference functions inside files in your new /data volume directory. This means you will need to change any path references to include /data, and specifically add the /data directory to your import path by adding this to the top of your IPython Notebook:

import sys
sys.path.insert(0, '/data')

## this relative dir won't work:
# data_dir = 'foo/dataset.1'
## so we just add /data to the front:
data_dir = '/data/foo/dataset.1' 

Finally, I try to be readable, but take a look at my earlier post linked above if you want a breakdown of what's going on in there. Have fun!

I've been using the fairly popular nginx-proxy reverse proxy for Docker containers, created by Jason Wilder. It's a slick, super-simple method of putting many containers on a single host that all need to share HTTP/HTTPS ports. I am also a huge fan of the Let's Encrypt project. Free SSL certificates as long as you can prove that you are the domain's operator. This is really how it should work: In my book, forcing people to pay for SSL certificates is shitty and exploitative, especially considering how incredibly important encryption is these days. And incidentally, the behavior around self-signed certificates in browsers is stupid and broken.

So anyways, I played with the Let's Encrypt stuff late last year when they went into public beta. Best Christmas present ever! So the idea is pretty simple: You tell your Let's Encrypt client (probably best to use certbot) that you want a certificate, and it talks to the Let's Encrypt server, requesting a certificate from the CA. That causes Let's Encrypt to make a curl call to your domain, requesting a specific resource that certbot creates. That resource looks something like http://www.example.com/.well-known/acme-challenge/g89SrgM4UAJGHiukm3GqQ3xMjTnpN-kZDYb27u4aTRW. That resource is just a regular file on disk. It has contents that look something like DTe7mGGhLlML7Vlh4dyNTu97OiIrIIs7xd5O0Fpmlq8.TaRs2K47il2D0K9RjmOKOx7Neuu91FdEpLp2Wo4FcNI. As long as that resource matches what Let's Encrypt is looking for, you get a free SSL certificate! And they've built some magic into certbot so that it automagically installs that cert into your webserver if it's a common one (e.g. Apache or Nginx or something).

I wanted to use it on on my Dockerized web frontends, which use nginx-proxy. I had spotted a couple of issues on nginx-proxy's Github page which mentioned Let's Encrypt, but I hadn't yet tried to get this working with my nginx-proxy container. Getting that to work does not seem like a trivial task. Not having the spare minutes to get Let's Encrypt working in my infrastructure, I put it on the back burner and made a mental note to check in every once in a while. Well, I completely forgot about it until I got an email recently reminding me to renew one of my SSL certs. And I'm not paying $8.99 for something that should be free, so I knew it was time to check back in with Let's Encrypt being incorporated into the nginx-proxy project.

Enter nginx-proxy-letsencrypt-companion. This is a docker container that sits coupled to your nginx-proxy container, sharing its volumes and paying attention to containers spinning up that have LETSENCRYPT_HOST and LETSENCRYPT_EMAIL environment variables set. The idea is that you start your nginx-proxy container, then start up this nginx-proxy-letsencrypt-companion container, and then start up your other containers that need Let's Encrypt certificates. The companion will request new Let's Encrypt certificates for containers that do not have current certificates and which also have those LETSENCRYPT_* environment variables set.

So here are my notes for getting this going. I ended up adding /usr/share/nginx/html as a data volume in my nginx-proxy container, and making a couple of the volumes rw instead of ro. Thus, my nginx-proxy run command looks something like this:

docker run -d \
    --name="nginx-proxy" \
    --restart="always" \
    -p 80:80 \
    -p 443:443 \
    -v "/var/docker/nginx-proxy/htpasswd:/etc/nginx/htpasswd" \
    -v "/var/docker/nginx-proxy/vhost.d:/etc/nginx/vhost.d" \
    -v "/var/docker/nginx-proxy/certs:/etc/nginx/certs" \
    -v "/var/run/docker.sock:/tmp/docker.sock" \
    -v "/usr/share/nginx/html" \
    jwilder/nginx-proxy

And the command for the brand new nginx-proxy-letsencrypt-companion container looks like this:

docker run -d \
    --name="nginx-proxy-letsencrypt-companion" \
    --restart="always" \
    -v "/var/run/docker.sock:/var/run/docker.sock:ro" \
    --volumes-from "nginx-proxy" \
    jrcs/nginx-proxy-letsencrypt-companion

And finally, your individual containers will follow this pattern (note the environment variables mentioned above):

docker run -d \
    --name="example.com" \
    --restart="always" \
    -e "VIRTUAL_HOST=example.com,www.example.com" \
    -e "VIRTUAL_PORT=2368" \
    -e "LETSENCRYPT_HOST=example.com,www.example.com" \
    -e "LETSENCRYPT_EMAIL=contact@example.com" \
    -v /var/docker/example.com/ghost:/var/lib/ghost \
    ghost

And there you have it! Once you get your containers up -- in that order: nginx-proxy, nginx-proxy-letsencrypt-companion, and your web container -- give the Let's Encrypt a minute to phone home and get a call back, and you'll have a free SSL certificate and related miscellany there in /var/docker/nginx-proxy/certs on your host. Also, the reason I created a data volume in the nginx-proxy container at /usr/share/nginx/html instead of a bind-mount volume. I do this because the Let's Encrypt client in the companion container will continually update the authorization data, and I don't want to have to worry later about cleaning up a huge, sprawling directory full of those files, some of which would contain valid information, and many others containing invalid information. Of course, you do whatever you want, as long as its right for you. Good luck!

Recently, I was offering some help on Stack Overflow to someone asking about deleting images from a private Docker registry. Here I mean a v2 registry, which is part of the Docker Distribution project. I should advise anyone reading this that no one ever refers to a v1 registry anymore: That project is dead, even though it occupies the registry:latest image tag on Dockerhub (you want to pull registry:2 at least). The v1 registry is an old Python project, and v2 is written in Go. Anyways, the v2 registry has not had delete capabilities (via the API) since its inception. This was my initial assumption, but I took the opportunity to research the latest information.

As it turns out, the latest versions of registry (later than v2.4 I think) do have delete functionality. While the main Docker project ("docker-engine") has excellent documentation, the Distribution project has previously not been. It's not bad, but it's not great, either. The API documentation is not very clear on using the new delete API functionality. But it's there, along with an interesting garbage collection mechanism. That's a topic for another day, but it's the reason I wanted to upgrade to version 2.4 of the registry. I was using version 2.1 or something.

Cue an upgrade, and a couple of problems. First, in the config.yml file, in the cache section under the storage section, the layerinfo setting has been deprecated and renamed to blobdescriptor. That isn't a blocking change yet, but it will be soon, so rename it now while you have the chance.

Finally, if you're backing your registry with S3 like a sane human being, the permissions have changed and the change is not documented anywhere. Zing! I couldn't push when I fired up my new registry container. I kept getting "Retrying in X seconds" messages when pushing individual layers. I killed and deleted the container, started up a new one with the level setting under the log section set to debug in my config.yml file. This yielded the key to the issue (notice the "s3aws: AccessDenied" message and 403 status code):

"err.code":"unknown","err.detail":"s3aws: AccessDenied: Access Denied\n\tstatus code: 403, request id: 11E0123C033B0DB5","err.message":"unknown error"

Here's what the new S3 policy needs to look like (beware copying from the documentation linked above: There is an errant comma in the documented policy):

 "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:ListBucket",
          "s3:GetBucketLocation",
          "s3:ListBucketMultipartUploads"
        ],
        "Resource": "arn:aws:s3:::mybucket"
      },
      {
        "Effect": "Allow",
        "Action": [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
          "s3:ListMultipartUploadParts",
          "s3:AbortMultipartUpload"
        ],
        "Resource": "arn:aws:s3:::mybucket/*"
      }
]

Just for the record, this new policy adds support for the s3.GetBucketLocation, s3:ListBucketMultipartUploads Actions on your particular bucket, and the s3:ListMultipartUploadParts and s3:AbortMultipartUpload Actions on your bucket contents.

So here's a fun surprise: I love machine learning just like everyone else! I have played around with a few ML concepts in a couple of weekend projects, but nothing serious. Earlier this year I came across a Coursera ML course which seemed like a great place to start a more formal education about the subject. Sweet! I'm a self-taught programmer, but taking university CS courses as a non-degree-seeking student helped fill the gaps in my knowledge. I see the value of formal education.

To prepare for the class, they want you to set up a Python 2.7 environment (Anaconda), IPython, and GraphLab Create. IPython is an interactive environment for programming languages. Obviously, it was first targeted for Python, but I guess they do all kinds of languages now. One of the cool features is built-in support for data visualizations. We're specifically concerned with the IPython "Notebook" feature set. We are also using GraphLab Create, which is a commercial product spawned out of a project from CMU and released by Dato. The CEO of Dato is one of the primary instructors of the course.

Real quick, GraphLab Create is a commercial product, as I've mentioned. Dato offers a free "student" license for this product, which is what we will be using for the course. This blog post assumes that you have already signed up for one of these educational licenses, and you have already been given your license key (which looks, for example, like ABCD-0123-EF45-6789-9876-54FE-3210-DCBA). You will need both this license key and the email address you signed up with in order to continue.

Now, all this software is great (really, you'll see what I mean when you start using it), but it seems to be quite a lot of stuff that I'd rather not have installed on my filesystem if I can help it. If you have spoken to me at all in the last two and half years, you know that I am a big fan of Docker. Probably 80% of this blog is about Docker. And so it should come as no surprise that I've got compartmentalization on my mind. I install and use just about everything Dockerized. These tools are phenomenally useful for this illuminating ML course, but I want them containerized.

I've taken the installation instructions for Anaconda Python and GraphLab Create and put them into a Dockerfile, which you'll have a chance to look at a little further down. Before I get to that, I want to point out that if you look closely at the install instructions for GraphLab Create, you'll see a mention of getting your Nvidia GPU to work with the software in order to speed things along. Specifically for machine learning, having a GPU workhorse can be a computational difference of days, weeks, or even years.

CPUs are fine for most projects and will probably work just fine for this course, but I have seen the question of CUDA processing specifically with respect to Docker. I had heard that CUDA was not available to Docker containers because of the difficulty of making the Nvidia drivers available to the containerized process. Well, I took it as an opportunity for research, and it just so happens Nvidia has very recently released an application called nvidia-docker specifically for making the GPU available to Docker containers! You can follow that link for all kinds of interesting information, but suffice it to say nvidia-docker is a drop-in replacement for the docker executable which is used on images that you want to be CUDA-capable. They also offer similar functionality in a daemon plugin called nvidia-docker-plugin. You can read about the differences in the nvidia-docker documentation.

A quick note about nvidia-docker: I ran into a problem with the .deb I had installed per their instructions, because I am using the latest version of Docker (1.11), and as of this writing, they haven't released an updated .deb with the working code. And so that meant that I had to complie my own nvidia-docker binary. It's super easy (and doubly so for anyone with the technical wherewithal to be taking a machine learning course): Just git clone https://github.com/NVIDIA/nvidia-docker and then cd nvidia-docker && make and then sudo make install and you've got a working binary!

Also, I ran into import errors during the ML course (specifically with matplotlib). I ended up solving this by installing python-qt4 inside the Docker container right off the bat.

So finally, here's what my Dockerfile looks like:

FROM ubuntu:14.04
MAINTAINER curtisz <software@curtisz.com>

# get stuff
RUN apt-get update -y && \
	apt-get install -y \
		curl \
		python-qt4 && \
	rm -rf /var/cache/apt/archive/*

# get more stuff in one layer so unionfs doesn't store the 400mb file in its layers
WORKDIR /tmp
RUN curl -o /tmp/Anaconda2-4.0.0-Linux-x86_64.sh http://repo.continuum.io/archive/Anaconda2-4.0.0-Linux-x86_64.sh && \
	chmod +x ./Anaconda2-4.0.0-Linux-x86_64.sh && \
	./Anaconda2-4.0.0-Linux-x86_64.sh -b && \
	rm ./Anaconda2-4.0.0-Linux-x86_64.sh
# make the anaconda stuff available
ENV PATH=${PATH}:/root/anaconda2/bin

## anaconda
RUN conda create -n dato-env python=2.7 anaconda
# (use JSON format to force interpretation by /bin/bash)
RUN ["/bin/bash", "-c", ". activate dato-env"]
RUN conda update pip

## install graphlab create with creds provided in --build-arg in 'docker build' command:
ARG USER_EMAIL
ARG USER_KEY
RUN pip install --upgrade --no-cache-dir https://get.dato.com/GraphLab-Create/1.9/${USER_EMAIL}/${USER_KEY}/GraphLab-Create-License.tar.gz

## install ipython and ipython notebook
RUN conda install ipython-notebook

## upgrade GraphLab Create with GPU Acceleration
RUN pip install --upgrade --no-cache-dir http://static.dato.com/files/graphlab-create-gpu/graphlab-create-1.9.gpu.tar.gz

CMD jupyter notebook

I ended up having to update this Dockerfile when Dato released GraphLab Create version 1.9. It was as easy as changing "1.8.5" to "1.9" in the Dockerfile. Everything else was the same. Keep this in mind if you find that you need to install a newer version of GraphLab Create. Now, you'll build the image with this command, making sure to replace the email and license key with your own details:

docker build -t=graphlab --build-arg "USER_EMAIL=genius@example.edu" --build-arg "USER_KEY=ABCD-0123-EF45-6789-9876-54FE-3210-DCBA" .

The build will take a few minutes. It downloads a few hundred megabytes of stuff. When you're done with that, you can launch IPython with the following command:

nvidia-docker run -d --name=graphcreate -v "`pwd`/data:/data" --net=host graphlab:latest

Voila! You've got this whole operation running and you can access your IPython notebook by going to http://localhost:8888/ in your browser!

Please note: When it comes time to use GraphLab Create, you will be able to browse its UI normally, because we have specified --net=host in the docker run command, which shares the host's network stack with the container. The reason we do it this way is because GraphLab Create uses tcp/0 to set its server port. If you remember, that means the system chooses a random high port number, which prevents us from targeting a specific port with an EXPOSE Dockerfile directive (or -p port assignment in the docker run command). Exposing the host's network stack to the container could have some security implications if you run an untrusted application in that container. The applications we're using for this course are okay, it's just something you should be aware of.

Finally, I've been asked to include a tiny Docker crash course in case it's new to you. So our particular run command also mounts the ./data/ directory into the container at /data! This means you can download notebooks and datasets for the course and put them in that directory, and they'll be accessible in the container under the /data directory. For example, you would use sf = graphlab.SFrame('/data/people-example.csv') to load the sample data. In your terminal, you can use docker logs graphlab to see the container's logs, but don't forget you can swap out -d with -it in your docker run command if you want to create an interactive session for the container so that you can see the output in your terminal. You can also drop into a shell on the running container with docker exec -it graphlab /bin/bash and poke around if you need to. Killing the container happens with docker stop graphlab and deleting the container happens with docker rm graphlab. The Docker documentation is generally well-written, concise, and accurate. The source is also very approachable, as is the Docker community itself! Don't be afraid to drop by #docker on Freenode IRC if you need help!

For your convenience, I have created a Github repository with the Dockerfile and related scripts, as well as the sample starter data provided by Coursera.

Good luck!

Recently I've had an interesting conversation in #docker on Freenode with a guy that's been trying to get crons working inside his Docker container. I hadn't yet had a chance to look at that, and so we took off on a late-night debug session exchanging Dockerfiles via Pastebin. He has a bunch of other stuff going on, but at the core, he's just running an Apache webserver instance and then wants to run some crons in that container as well. I took his Dockerfile and related scripts, and pared them down to the bare minimum, commenting out everything that wasn't related directly to getting Apache and cron to work. You can take a look at what I came up with:

FROM ubuntu:14.04
MAINTAINER curtisz <software@curtisz.com>

# we install stuff this way to keep it all on one layer
# (which reduces the overall size of our image)
RUN apt-get update -y && \
	apt-get install -y \
		cron \
		apache2 \
		supervisor && \
	rm -rf /var/lib/apt/lists/*

# apache stuff
RUN mkdir -p /var/lock/apache2 /var/run/apache2 /etc/supervisor/conf.d/
ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2
ENV APACHE_LOCK_DIR /var/lock/apache2
ENV APACHE_PID_FILE /var/run/apache2.pid
# empty out the default index file
RUN echo "" /var/www/html/index.html

# cron job which will run hourly
# (remember that COPY is better than ADD for plain files or directories)
COPY ./crons /etc/cron.hourly/
RUN chmod +x /etc/cron.hourly/crons
# test crons added via crontab
RUN echo "*/1 * * * * uptime >> /var/www/html/index.html" | crontab -
RUN (crontab -l ; echo "*/2 * * * * free >> /var/www/html/index.html") 2>&1 | crontab -

# supervisord config file
COPY ./supervisord.conf /etc/supervisor/conf.d/supervisord.conf
 
EXPOSE 80
WORKDIR /var/www/html/
CMD /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

You can tell what's going on here, it's pretty straightforward. To get this working, we need to install apache2, supervisor, and of course, cron. The next few lines are configuration options for Apache. Then finally we get to the test crons. I'm dumping a simple cron shell script into /etc/cron.hourly in the container and making it executable, and then creating two new crons via crontab. Please note how I have added these crons with crontab - by also listing the previously-added crons with crontab -l. If you don't do this, whatever you add via crontab will overwrite whatever's in there now. My crons are just stupid simple crons for dumping something easy into Apache's index.html file so we can prove they're running. My little crons file looks like this:

#!/bin/sh
ps aux | grep apache > /var/www/html/index.html

Some things to notice about this file... First, it starts with a typical shell script shebang (#!/bin/sh). It should also be executable. Lastly, this file can only contain alphanumeric characters in its filename (and underscores). It cannot contain a dot, which means naming it something like crons.sh won't work.

Next, we're adding the config file for supervisord. Then simply exposing the port, changing our working directory, and then specifying our CMD to start supervisord. Speaking of, this is what our supervisord.conf file looks like:

[supervisord]
nodaemon=true
logfile = /var/log/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
 
[program:cron]
autorestart=false
command=cron -f
 
[program:apache2]
autorestart=false
command=/usr/sbin/apache2ctl -D FOREGROUND

Pretty standard fare. And so let's build our container and start it:

docker build -t="crontest" .
docker run -it --name="crontest" -p 8080:80 crontest:latest

And there you have it! Give your crons a few minutes to execute, then fire up a browser on your localhost and point it to http://localhost:8080/index.html. You should see the output of our test crons there at the tail of the file. Refresh to see more.

That's all there is to it! Previously, I've used the cron system available on the Docker host, which certainly has its benefits. First and foremost of which is not having to use supervisord inside a Docker container. Since Docker is just a fancy way to run a process, you want to avoid loading up your container with a bunch of cruft. It's not a VM! But when you can't avoid it and you really need to run crons alongside your containerized processes, it's no sweat to get it going.

Today in #docker on Freenode there was a person with a problem with their v1 Docker registry. I think I jinxed it when I said it was "extremely easy" to get a v2 registry running behind an Nginx proxy. It turned into a nightmare, and I'm sharing the design process to help anyone else that might need to debug problems with a similar setup.

So I had previously spent time getting a private registry to work behind the jwilder/nginx-proxy image, which is a great reverse proxy for docker containers. There is a lot of movement with Docker and especially the ecosystem of orchestration applications that exist around it. These days, plenty of stuff out there does what the nginx-proxy image does. Personally I like Consul and Serf from Hashicorp. Incidentally, they also make a hell of a nice application -- Vault -- that solves most of the problem with sharing sensitive configuration information. Anyways, for single-host-multiple-container environments, I still prefer proxying with the nginx-proxy image. I use it to front for all my web applications, and our Gitlab installation, so it only made sense to front my v2 registry with the same proxy.

The way the nginx-proxy image works is that it is bound to listen on tcp/80 and tcp/443 and uses the host machine's /var/run/docker.sock to listen for docker events, and then other containers are started with a VIRTUAL_HOST environment variable. When that happens, the docker-gen utility creates an Nginx template and starts routing requests to the container. You can also mount htpasswd files into the proxy and SSL certs to manage authentication and HTTPS connections. It also supports custom directives and custom templates.

So anyways, the problem the guy was having I was pretty sure he wouldn't be able to get support for, because v1 has been officially deprecated on Docker Hub, and also is no longer the primary registry endpoint the Docker client looks for when attempting to connect to a registry. Support for v2 was introduced in 1.6, and as of version 1.9, the client now prefers v2 registry endpoints over v1. So the time has come to upgrade.

Let's get started. First, pull the images we need to work with:

docker pull jwilder/nginx-proxy
docker pull registry:2.2

It's important to note that registry:latest does not point to the latest version of registry. The latest version points to v1! We need to make sure we're pulling v2. The docker registry is actually a part of the "distribution" repository. You can and should check for the latest version of both the v2 registry image and also any documentation there. It's generally true that the Docker documentation is mostly very good and correct, so make sure you read that and prefer that information above mine.

To configure the v2 registry, we need to create a minimal config.yml file. I usually keep all my stuff for docker under /var/docker/<container>, so sudo sh -c "mkdir -p /var/docker/registry && cd /var/docker/registry/ && vim config.yml" and put this into it:

version: 0.1
log:
    level: info
    formatter: json
    fields:
        service: registry
        environment: staging
        source: registry
http:
    addr: :5000
    host: myregistry.example.com
    secret: biglongsecretwhatever
storage:
    filesystem:
        rootdirectory: /var/lib/registry

This is the bare minimum you'll need to get your registry going. You should change http.secret to something long and random. For example you can use a bash one-liner like cat /dev/urandom | tr -dc a-zA-Z0-9 | head -c36 to get a random string. Now save this yaml file. Finally, mkdir /var/docker/registry/lib to create a directory for storing our registry images on the local host. I like to use AWS S3, and if this interests you, take a look at my last post on this subject for instructions.

We've got our v2 registry primed to run, but we won't run it quite yet.

Let's get the nginx-proxy image going. First mkdir /var/docker/nginx-proxy && cd /var/docker/nginx-proxy to get into our base host directory. Use mkdir vhost.d to create a directory for storing our special Nginx directives.

Now, if you want to have nginx-proxy handle your SSL certificates and authentication -- and we do, since docker will complain about a registry running without HTTPS -- you'll want to mdir htpasswd && mkdir certs.d at this time as well. For illustrative purposes, let's say our registry domain is myregistry.example.com and we've already pointed DNS at the host. So we'll name our SSL certificate to myregistry.example.com.crt and our key to myregistry.example.com.key and drop both of those files into the /var/docker/nginx-proxy/certs.d directory.

With HTTP authentication, we can just do some real basic stuff. We don't need anything fancy, since the client supports basic authentication. You can use htpasswd (from the apache2-utils repo on Linux Mint or Ubuntu) to generate authentication information. Save this information in /var/docker/nginx-proxy/htpasswd/myregistry.example.com similar to the way we named our SSL data. For future reference, you can also store a default certificate and key for HTTPS requests that arrive at your nginx-proxy which aren't routable to one of your containers.

We need a couple more files dropped into /var/docker/nginx-proxy/vhost.d. The first is myregistry.example.com and looks like this:

client_max_body_size 0;
chunked_transfer_encoding on;
 
location /v2/ {
  # Do not allow connections from docker 1.5 and earlier
  # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
  if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
    return 404;
  }
 
  add_header Docker-Distribution-Api-Version "registry/2.0";
  #more_set_headers     'Content-Type: application/json; charset=utf-8';
  include               vhost.d/docker-registry.conf;
}
 
location /v1/_ping {
  auth_basic off;
  include               vhost.d/docker-registry.conf;
  add_header X-Ping     "inside /v1/_ping";
  add_header X-Ping     "INSIDE /v1/_ping";
}
 
location /v1/users {
  auth_basic off;
  include               vhost.d/docker-registry.conf;
  add_header X-Users    "inside /v1/users";
  add_header X-Users    "INSIDE /v1/users";
}

These directives do a couple of things. First, they lift the artificial limit Nginx places on request size. Since you're going to be uploading huge files to your v2 registry, we need to turn off that limitation. This directive also prevents access by client versions 1.5 and below (which only use v1 registry endpoints anyway). Of particular interest is that the location directives for v1 endpoints have been proven to fix docker client bugs which caused the v2 registry to throw a 404 when trying to connect.

The second file is /var/docker/nginx-proxy/vhost.d/docker-registry.conf and looks like this:

proxy_pass                          http://myregistry.mexample.com;
proxy_set_header  Host              $http_host;   # required for docker client's sake
proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
proxy_set_header  X-Forwarded-Proto $scheme;
proxy_read_timeout                  900;

This forwards the IP address of your clients to the registry for logging purposes, as well as providing a couple of required headers.

Now that we've configured our v2 registry and nginx-proxy, let's start them up! We'll begin with the nginx-proxy container (shoutout to Arthur for noticing I'd forgotten to mount vhost.d in this next command):

docker run -d \
  --name "nginx-proxy" \
  --restart "always" \
  -p 80:80 \
  -p 443:443 \
  -v /var/docker/nginx-proxy/certs:/etc/nginx/certs:ro \
  -v /var/docker/nginx-proxy/htpasswd:/etc/nginx/htpasswd:ro \
  -v /var/docker/nginx-proxy/vhost.d:/etc/nginx/vhost.d \
  -v /var/run/docker.sock:/tmp/docker.sock \
  jwilder/nginx-proxy

Next, let's start our v2 registry container:

docker run -d \
  --name="registry" \
  --restart="always" \
  -v "/var/docker/registry/config/config.yml:/etc/docker/registry/config.yml" \
  -v "/var/docker/registry/lib:/var/lib/registry" \
  -e "VIRTUAL_HOST=myregistry.example.com" \
  registry:2.2

Notice here that we're not binding our v2 registry port to any port on the host. This is because we don't want to bind that port to the host interface. We want to "bind" that port to the nginx-proxy container. You may be wondering how the nginx-proxy container knows that it should route inbound HTTPS requests to port 5000 on the registry container. The answer is that it binds to whatever port your container has exposed, either via EXPOSE in the Dockerfile or --expose in the docker run command. In our example, the v2 registry image has EXPOSE 5000 in its Dockerfile.

This brings me to one last pain point I had with the nginx-proxy image. As wonderful as it is, it's not clear that you cannot proxy anything that isn't based on the HTTP protocol. So for example you can't have the nginx-proxy image front for your SMTP server or your FTP server. Also, something else that took me a while to understand... Let's say you want to run a cAdvisor container to expose some metrics for your Prometheus server. The documentation wants you to make that thing listen on tcp/8080. That's totally fine, but if you want to run this container behind your nginx-proxy container, you'll want to not bind that port to the host. And since it's exposed port 8080 in its Dockerfile, you can simply start it with -e "VIRTUAL_HOST=cadvisor.example.com" and it will then be available at https://cadvisor.example.com/, which will be served from behind your nginx-proxy. The nginx-proxy container gets an inbound HTTPS request on tcp/443, then routes the connection on the backend to tcp/8080 on your cAdvisor container. No --link necessary!

A piece of advice when debugging problems: Start with docker logs nginx-proxy and check if the problem is with your container or if it's with your nginx-proxy. Also you can use docker exec -it nginx-proxy /bin/bash to drop into a shell on your nginx-proxy container and poke around. You should check the /etc/nginx/conf.d/default.conf to make sure your container is being properly exposed to the nginx-proxy and its docker-gen utility.

Good luck!

A friend of mine recently asked me about the publish/subscribe ("pubsub") programming pattern. As this is something I use in almost every project, I thought I'd be able to find a decent tutorial for him online. Something that would be helpful to someone familiar with programming, but not familiar with this pattern. As it happens, most of the pubsub documentation or tutorials out there are specific to their use in one situation: APIs. That's all fine and dandy, but the pubsub pattern is so much more powerful and applicable in so many more situations than that niche. I am pretty sure it's my favorite programming pattern of them all. Not so hard to believe once you know that I write JavaScript for a living. But, I think JavaScript programmers are more sensitive to the applicability of the pubsub pattern than programmers in other languages, because JavaScript is asynchronous.

So what exactly does that mean?

If you've ever written any JavaScript, you have probably gotten yourself in trouble with its asynchronous nature. For example, you've probably thought about doing something like this:

// read data from a file ...
var data = readDataFromFile('/path/to/file.txt');
// ... then do something with the data
console.log('your name is: ' + data.username);

But you can't do that. JavaScript's asynchronous nature means the (totally made-up) function readDataFromFile('/path/to/file.txt') is not executed immediately. Instead it is tossed onto the event queue and executed later, after the console.log('your name is: ' + data.username); is executed. This happens because all the code in the current scope is executed in a batch, and then JavaScript grabs the next thing on the event queue and executes everything in that function's scope, and so on. "External" function calls get put onto the event queue as they are encountered, instead of executed, as they are in synchronous languages like Python or C. You could write a lot of JavaScript before you encounter this behavior, though. JavaScript is pretty nebulous about when functions get put on the event queue versus executed directly. Usually you trigger the event queue when making calls out to external resources, for example via AJAX or file I/O. Well, in order to work around this limitation in JavaScript, we use callbacks. Callbacks are functions that we pass as parameters to other functions that get executed after all the processing inside the first function is done. That sounds stupid, so let me illustrate. Here's an example of proper asynchronous JavaScript using callbacks:

// read data from a file ...
fs.readFile('/path/to/file.txt', function( err, data ) {
    //                           ^^^^^^^^^^^^^^^^^^^^^
	// ... then do something with the data *in a callback*
	if (err) {
    	throw new Error('problem doing stuff: ' + err);
    }
    console.log('your name is: ' + data.username);
});

Now you're probably asking yourself why I'm even talking about asynchronous programming in JavaScript. What does that have to do with the pubsub pattern? Well, after a while, you fall into what's called "callback hell" when you chain callbacks with callbacks with callbacks. Take a look at what should be a simple operation: Reading from a file, making a change to the data, then writing those changes to a file:

var filename = '/path/to/file.txt';
var savefile = '/path/to/new/file.txt';

fs.stat(filename, function(err, stat) {
	// inside first callback
	if (err) throw new Error('could not stat file: '+err);
    if (!stat.isFile()) throw new Error('path is not a file!');
    
    fs.readFile(filename, function(err, data) {
    	// inside second callback
    	if (err) throw new Error('could not read file: '+err);
        
        modifyUsername(data.username, function(name) {
        	// inside third callback
            
        	fs.stat(savefile, function(err, stat) {
            	// inside fourth callback
            	if (err) throw new Error('could not stat file: '+err);
                if (!stat.isFile()) throw new Error('path is not a file!');
                
                fs.writeFile(savefile, name, function(err) {
                	// i want to kill myself ..............
                    // and look at all the close brackets and parentheses!
                    // how embarrassing!
                });
            });
        });
    });
});
// yikes. are you sure you closed up all your functions properly?

"Holy shit. JavaScript sucks!" they will say. And so. Even the most faithful will pause to think.

Now, if only there were a way to write a blob of callback code, and then create a "trigger" that we could pull when we were ready to execute the callback blob, we could clean this mess right up. ... Well, that's right! You've figured out that my beloved pubsub can swoop in and save the day.

Here is what super-simplified pubsub calls look like:

// when you have a blob of code you want to run later:
subscribe(  "this-can-be-any-string",    functionToCall    );
// then when you're ready to execute the blob of code:
publish(  "this-can-be-any-string",   [  array, of, parameters  ]);

The first parameter to the subscribe() function is a string that we use to "index" the function we want to call. We store the function using this string as a label. This is so that later, when we call the publish() function, we reference the function we want to execute using the same string ("this-can-be-any-string") and then the second parameter passed to the publish() function is an array of arguments to pass to the function we will execute. In the above two lines, the result of the code ends up logically looking like this:

functionToCall(array, of, parameters);

Take a look at the pubsub object. It's very easy to read:

// here is our pubsub object
// this is how we enabled the pattern
var $pubsub = (function() {
	var cache = {};
    function _flush() { cache = {}; }
	function _pub( topic, args, scope ) {
    	if (cache[topic]) {
        	var current = cache[topic];
            for (var i=0; i<current.length; i++) {
				current[i].apply(scope || this, args || []);
            }
        }
    }
    function _sub( topic, callback ) {
    	if (!cache[topic]) {
        	cache[topic] = [];
		}
        cache[topic].push(callback);
    }
	return {
    	flush: _flush,
    	pub: _pub,
    	sub: _sub
    };
})();

Now take a look at the refactored code, which uses pubsub to escape callback hell by "subscribing" some functions to events which we later "publish":

// convenience function to DRY up our calls to fs.stat()
var fileStat = function( filename, callback ) {
	fs.stat(filename, function(err, stat) {
    	if (err) throw new Error('could not stat file: '+err);
        if (!stat.isFile()) throw new Error('path is not a file!');
        typeof(callback) === 'function' && callback();
    });
};
// now begins our list of discrete functions to execute in a specific order
// (note the $pubsub.pub() calls within each function!)
var fileRead = function( filename ) {
	fileStat(filename, function() {
        fs.readFile(filename, function(err, data) {
        	if (err) throw new Error('could not read file: '+err);
            $pubsub.pub('/username/modify', [data]);
        });
    });
};
var fileWrite = function( filename, data ) {
	fileStat(filename, function() {
        fs.writeFile(filename, data, function(err) {
        	if (err) throw new Error('could not write file: '+err);
            $pubsub.pub('/continue/process');
        });
    });
};
var modifyUsername = function( username ) {
	var newUsername = username + '_modified';
	$pubsub.pub('/file/write', ['/path/to/new/file.txt', newUsername]);
};
var moreStuff = function() {
	// do more stuff after everything else
};

// set up our subscriptions
$pubsub.sub('/file/read', fileRead);
$pubsub.sub('/file/write', fileWrite);
$pubsub.sub('/username/modify', modifyUsername);
$pubsub.sub('/continue/process', moreStuff);

// kick off the whole process with this "publish" statement
$pubsub.pub('/file/read', ['/path/to/file.txt']);

This looks a lot nicer, doesn't it? It's a bit more typing, but once you grok what's happening here in this post, you will never want to go back to that awful callback hell. So I've spent the last few minutes answering the 5 Ws of the publish/subscribe pattern by showing you how to do it. If you're not totally clear on what's happening here, start reading from the top, and hand-copy the code into your IDE. Taking a closer look by hand-copying the code is something that always helps things sink in. Before you know it, you'll grok the pubsub pattern and be using it to hit all kinds of nails.

One of the members of an ARG I play recently started talking about an old piece of equipment he'd purchased, which supposedly had been used by phone repair technicians to do their work. The equipment in question is an Itronix T5000, which has an in-built modem, speedy 486 processor, and 640KB of RAM. Kilobytes, folks. This was the 90s. You know, incidentally, I fondly remember having 640KB of RAM in my very first computer, and having to juggle peripherals.

Anyways, unfortunately for our friend, when he powered the device on, this is what he saw:

Our friend Mister Argent managed to offload all of the device's files to USB using an available restore feature. He just didn't know how to proceed. Fortunately for him, when it comes to binaries from the 90s, I'm your guy. Now, it's been a few years since my last encounter with something of this nature, and I hadn't realized that most of the tools we used to use for reversing no longer function on today's platforms. A simple strings of the files gave me nothing useful. Interesting, sure, but nothing useful to solve our primary predicament. There is a PASSWORD.DAT file in the collection but it's clearly not plaintext and here specifically, I am nothing more than a hobbyist, and definitely no cypherpunk. I would need to reverse this binary to get anywhere. It didn't take me long, however, to remember the only game I spend time playing these days -- which also happens to be a binary from the 90s -- and more importantly, the platform I use to play it: DOSBox.

If you've never heard of DOSBox, it is basically what you're thinking it is after my description above: An emulator for DOS applications. The thing about DOSBox that makes it special -- besides being the key to many glorious, wonderful games from the 90s that you couldn't otherwise play -- is that the creator has built in a very useful debugger.

I'm a die-hard Linux Mint user, since I can't stand Ubuntu's Unity UI almost as much as I can't stand Michael Shuttleworth. One of the nice things about Linux Mint -- besides the fact that it hasn't immediately jumped into the systemd assimilation chamber -- is that it uses Ubuntu as a base, and therefore has its repositories available for consumption. DOSBox is available in Ubuntu's default repository (and probably in other default distro repos), but if you want to use the debugger, you've got to compile it with a special option, which means building from source. On LM/Ubuntu, you're going to need a few things in order to compile. If you're adventurous, you probably already have build-essential, autoconf, and automake. If not:

sudo apt-get install build-essential autoconf automake

Either way, you're going to need to get the DOSBox dependencies:

sudo apt-get build-dep dosbox

When you're compiling DOSBox for its debugger, you need a curses library. You'll need to install one to continue. Thanks to this answer on Stack Overflow, resolving this on Linux Mint/Ubuntu is a cinch:

sudo apt-get install lib32ncurses5-dev

Next, download the source and extract the source (sorry for the SourceForge link):

wget "http://downloads.sourceforge.net/project/dosbox/dosbox/0.74/dosbox-0.74.tar.gz"
tar -xvf dosbox-0.74.tar.gz

Next we'll build our awesome DOSBox debugger, but something you should know here is that DOSBox actually comes with two "levels" of debugging capability. Compiling with --enable-debug will get you most of the debugging features, but there are a few important ones you'll want the convenience of by compiling with --enable-debug=heavy. Most importantly, this "heavy" debugger enables the heavycpu command, which is a hardcore CPU logger that makes following code a lot easier:

cd dosbox-0.74
./autogen.sh

At this point, we're going to need to modify the source a little bit to prevent some errors in the actual compilation. Thanks to this helpful post by the DOSBox author, we know exactly what we need to change in the source to prevent the error. Let's create a little patch file to do the work for us. Create a new file in the dosbox-0.74 directory:

vim ./dosbox-0.74.patch

Paste the following contents into the editor:

diff -rupN dosbox-0.74/include/dos_inc.h dosbox-0.74.patched/include/dos_inc.h
--- dosbox-0.74/include/dos_inc.h	2010-05-10 10:43:54.000000000 -0700
+++ dosbox-0.74.patched/include/dos_inc.h	2015-07-07 14:52:42.057078234 -0700
@@ -28,6 +28,8 @@
 #include "mem.h"
 #endif
 
+#include <stddef.h>
+
 #ifdef _MSC_VER
 #pragma pack (1)
 #endif
diff -rupN dosbox-0.74/src/cpu/cpu.cpp dosbox-0.74.patched/src/cpu/cpu.cpp
--- dosbox-0.74/src/cpu/cpu.cpp	2010-05-12 02:57:31.000000000 -0700
+++ dosbox-0.74.patched/src/cpu/cpu.cpp	2015-07-07 14:52:23.641077942 -0700
@@ -30,6 +30,7 @@
 #include "paging.h"
 #include "lazyflags.h"
 #include "support.h"
+#include <stddef.h>
 
 Bitu DEBUG_EnableDebugger(void);
 extern void GFX_SetTitle(Bit32s cycles ,Bits frameskip,bool paused);
diff -rupN dosbox-0.74/src/dos/dos.cpp dosbox-0.74.patched/src/dos/dos.cpp
--- dosbox-0.74/src/dos/dos.cpp	2010-05-10 10:43:54.000000000 -0700
+++ dosbox-0.74.patched/src/dos/dos.cpp	2015-07-07 14:52:11.929077757 -0700
@@ -31,6 +31,7 @@
 #include "setup.h"
 #include "support.h"
 #include "serialport.h"
+#include <stddef.h>
 
 DOS_Block dos;
 DOS_InfoBlock dos_infoblock;
diff -rupN dosbox-0.74/src/ints/ems.cpp dosbox-0.74.patched/src/ints/ems.cpp
--- dosbox-0.74/src/ints/ems.cpp	2010-05-10 10:43:54.000000000 -0700
+++ dosbox-0.74.patched/src/ints/ems.cpp	2015-07-07 14:51:59.081077554 -0700
@@ -32,6 +32,7 @@
 #include "setup.h"
 #include "support.h"
 #include "cpu.h"
+#include <stddef.h>
 
 #define EMM_PAGEFRAME	0xE000
 #define EMM_PAGEFRAME4K	((EMM_PAGEFRAME*16)/4096)

Now save the file and exit, and apply the patch you have just created:

patch -p1 < ./dosbox-0.74.patch

Finally, compile DOSBox:

./configure --enable-debug=heavy
make

I've already got DOSBox installed, and so I chose not to install over it with my debugger-enabled version. But if you don't care, go ahead and place your newly-built debugging DOSBox version into your executables directory:

sudo make install

Awesome! We've got a bitchin' debugger! The second part of this story will cover the password discovery process using our fresh-from-source DOSBox debugger.

All I need is a private Docker registry that I can host myself.

If you're anything like me, you've been excitedly awaiting the release of the v2.0 Docker Registry. Version 1 was not very good. The company behind Docker is in no hurry to bite the hand that feeds them, and so development of the registry has been spotty at best. Among other things, the documentation is not great and the registry has no built-in authentication protocol. I understand that it's much better for business to get people frustrated with setting up their own private registry and then point them at your hosted services, where it's very easy to write a check to have someone else take care of this mess for you. But I am not in the habit of writing checks, and my check would probably bounce anyways.

The documentation available for deploying a v2 registry is specific to one situation. It is a set of instructions for using Compose (yet another Docker technology with a seemingly nebulous purpose at this time) to get both a v1 and a v2 registry working behind an Nginx proxy. But as I am using the Nginx Docker reverse proxy by Jason Wilder, so I don't need to bring in an external Nginx server. Nor do I need to answer requests for a v1 registry, as I am not using any Docker clients earlier than version 1.6.0.

All I need is a private Docker registry that I can host myself.

So... What we need to do is rip out all the extra stuff so that we're left with what we need. We don't need a v1 registry, so ignore all that. We're using jwilder/nginx-proxy to proxy our inbound requests, so ignore the instructions about pulling in the Nginx server. The average docker user right now doesn't really know what Compose is for or what it does -- though it will reduce complexity for most of us some day in the future -- so just ignore all the cruft about Compose. We're left with something close to what we're looking for.

First, clone the Distribution repository from Github and change into that directory:

git clone https://github.com/docker/distribution && cd distribution

We'll build our registry server from this repository:

docker build -t=registryv2 .

Now our container will build and should be listed in the output of docker images. We're almost there, but first we need to configure the registry. The future may include something I've talked about before called the docker vault, which is a cryptographically-secure, in-container, ephemeral storage mechanism which holds our sensitive configuration data. But for the moment, we don't have access to the vault because it doesn't exist. Here, we're going to have to rely on storing configuration data in a file, and then mount a volume on the host which exposes our config file to the container.

The v2 registry currently reads configuration data from cmd/registry/config.yml, so we need to map a directory on the host to this directory in the container. If you're trying to configure a v2 registry, just totally forget about everything related to configuring a v1 registry. The new configuration options are in the documentation. I'll include my own sanitized debug mode configuration file so you've got a sanity check reference:

version 0.1
log:
    level: debug
    fields:
        service: registry
        environment: staging
storage:
    filesystem:
        rootdirectory: /registry
    cache:
        layerinfo: inmemory
http:
    addr: :5000
    secret: somerandomstring
    debug:
        addr: localhost:5001

This is all pretty self-explanatory, if you've played with running any version of a docker registry. I'm running this on tcp/5000 during development for testing purposes. We are also running the server to listen on tcp/5001 (on local loopback only) if we need to connect and get some verbose debug information.

Now for production mode configuration, I use a different setup, which is more like something you'd expect to see out there in the real world:

version 0.1
log:
    level: info
    fields:
        service: registry
        environment: staging
storage:
    s3:
        accesskey: AKIA0Z6307DRPWJ5VH03F
        secretkey: OgP2Yhk1ZjFFf+aYokvnqI3qTlenCxSW2nbb9zpB
        region: us-west-1
        bucket: example.com-docker-registry-v2
        encrypt: false
        secure: true
        v4auth: true
        chunksize: 5242880
        rootdirectory: /registryv2
    cache:
        layerinfo: redis
http:
    addr: :443
    secret: ZpAedwVDFHK7mkNFFKSP8OQY
    debug:
        addr: localhost:5001
redis:
    addr: localhost:6379
    db: 0

Here, we're setting our registry to use an AWS S3 backend (that configuration data is of course dummy data, but feel free to try it, leet). We're also using a Redis container for caching, which speeds things up considerably. Again, the jwilder/nginx-proxy container auto-detects which port is exposed on a container, and I want this registry to listen on HTTPS (tcp/443), so I've changed its listen port appropriately.

So, because our registry is going to use Redis for caching, we need to spin up a Redis instance and link it to our registry container. Real quick, let's pull that Redis image:

docker pull sameersbn/redis

Now run it:

docker run -d --name="registryv2-redis" --restart="always" sameersbn/redis

And when I run my registry container, it looks something like this:

#!/bin/bash
docker run -d \
  --name="registryv2" \
  --restart="always" \
  --link registryv2-redis:redis \
  -v /var/docker/registryv2/config:/go/src/github.com/docker/distribution/cmd/registry \
  registryv2

Finally, it's not clear in the documentation, but to create a repository on your new v2 registry, you've got to tag your images correctly. Let's suppose for a second you have just created an image called myContainer and you'd like to create that on your new v2 registry:

docker tag myContainer:latest registry.example.com:443/myContainer:latest

This command tags your container not only with the latest tag, but also specifies exactly which registry you want to use for your new repository. Now you can push this image to your new v2 registry. Enjoy!

The thoughtful engineering behind the Twelve-Factor App design process has pushed environment variables as the place for sensitive configuration data such that it's somewhat of a standard these days. I have spent a couple of years now using environment variables to store sensitive configuration data in order to compare it to using time-tested configuration files. I think finally I have decided that configuration files are the way forward.

This gist summarizes fairly well the pros and cons of using environment variables. If you aren't familiar with the argument, or haven't given it much thought, I recommend you take a minute to read it over.

When I first heard of using environment variables for sensitive data, I was intrigued, but also initially skeptical. I've been doing this stuff for years and years. If it ain't broke, don't fix it, but... I'm always in the market for better solutions, so I didn't want to reject it forthright. I wanted to give it time, really roll the concepts around in my head (and my infrastructure) before making the call.

A lot of the opposition to config files comes from arguing that a deployment should scrub the sensitive data out of the environment once it's finished using it. However, in my experience environment variables are unintentionally "leakier" than files on the filesystem. Even if you scrub the environment, this data could live on in process lists and temporary files and other little nooks and crannies which are hard (or impossible) to get to, especially for the average person deploying things. With files, there's no question where your sensitive data lives, and this makes it easier to handle and to quarantine, even if only conceptually.

My specific situation is with loading docker containers with the sensitive data, and there's been some intriguing discussion among core devs about a kind of "docker vault" in #docker-dev on IRC. The mechanic they want to invent is a cryptographically-secure, ephemeral storage area where sensitive configuration data can be injected into a container, used, then destroyed. This will solve a lot of the issues associated traditionally with this specific problem. However until the time that arrives, I'm sticking with configuration files.

Hello there! This talk was posted on HN the other day, and I think it's excellent enough to post here.

There's a fascinating post I read recently that talks about skill progression, and how it relates to your perception of your own skill level, your frustration, your perception of the skill itself, and your actual abilities in a skill. I could talk on and on about how this relates to this skill, or that, and how to maximize the upward trend. But I won't do nearly as good a job explaining as this image from the original post:

Isn't that amazing insight? You can abstract this graph to represent virtually any technical skill you can think of. Replace the word "painting" with "programming" or "playing chess". Almost any technical skill fits this progression model.

I am still in a state of shock over Tom Wheeler's comments over at Wired. It looks like he's going to recommend essentially what we all thought was impossible: Title II regulations for broadband providers, including mobile broadband providers!

At first blush, the situation seems dire. Tom Wheeler was a top cable industry lobbyist before he took the position of chairman of the FCC. It doesn't take a lot of explaining for anyone to see that this is a pretty clear conflict of interest. It's as insane as Dick Cheney having to quit his job as CEO of Halliburton in order to become the Vice President of the United States. It shouldn't surprise anyone that the next eight years were full of absurd government spending on no-bid contracts given to -- naturally -- Halliburton. So it seemed the future of the internet would be like a Comedy Central show whose humor is based solely on schadenfreude.

However... Something happened. I'm intensely curious about the reason Chairman Wheeler decided to bring Title II to broadband. Because whatever it is, it's miraculous. He's almost certainly slammed the revolving door shut on himself. He's single-handedly responsible for recommending Title II for broadband providers, and for him to include mobile providers (i.e. cellphone network providers) as covered under Title II regulations is a huge blow to the zero-ethics, sociopath entities like Time Warner Cable, Comcast, and Verizon, who fight so hard to screw consumers for whatever profit they can manage.

It almost seems too good to be true. We aren't given too many things in our lifetime that are "too good to be true". I'm still cautious and thinking about what this could mean. I want to take it at face value and rejoice in what seems to be profound evidence that people in our government still care about protecting its citizen consumers. We'll just have to wait and see. We've still go to get this recommendation past the rest of the committee (which seems likely, despite there almost certainly being dissent among some members). The vote is February 26th. There are also some very smart net neutrality folks looking into the future at the possibility that Congressional Republicans will try to revoke the Telecommunications Act, thereby skirting the FCC as an entity with the power to create and enforce these kinds of regulation.

For the moment, however, the future looks bright.