CoreOS: Etcd in Docker containers

At &yet we have become big fans of CoreOS and Docker, and we've recently moved a bunch of our Node applications there.

We wanted to share a little tidbit for something that seems like it should be easy, but ended up being a minor stumbling block: making CoreOS's Etcd cluster available in Docker.

Here's a little background for those not up to speed on CoreOS or Etcd. CoreOS bills itself as "Linux for Massive Server Deployments." CoreOS is usually deployed with Etcd, "a highly-available key value store for shared configuration and service discovery," giving servers in a cluster the ability to share configuration data. We have been using Etcd and Confd for some time, so we naturally thought: "wouldn't it be cool to configure applications and services within Docker from Etcd?"

This can be easily accomplished by passing an environment variable to Docker at runtime and tweaking a few iptables rules. CoreOS makes the Etcd HTTP API available on its private IP address at port 4001 by default. CoreOS also sets an environment variable called $COREOS_PRIVATE_IPV4 in /etc/hosts with said IP. Fortunately, Docker is networked to the host's private IP, so this is the easiest point of access for Etcd.

Here's a sample unit file for Fleet and Dockerfile with the needed configuration to bring it all together:

Unit File

[Unit]
Description=Node App
Requires=docker.service

[Service]
EnvironmentFile=/etc/environment
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill node-app
ExecStartPre=-/usr/bin/docker rm node-app
ExecStartPre=/usr/bin/docker pull {{docker_registry}}/node-app
ExecStart=/usr/bin/docker run --rm=true --name=node-app -P -e COREOS_PRIVATE_IPV4=${COREOS_PRIVATE_IPV4} registry/node-app
ExecStop=/usr/bin/docker stop node-app

There are two relevant parts of the unit file to make it all work. First, EnvironmentFile=/etc/environment must be set, otherwise ${COREOS_PRIVATE_IPV4} will be empty. The second is setting -e COREOS_PRIVATE_IPV4=${COREOS_PRIVATE_IPV4} so your container also has access to the private IP.

Dockerfile

FROM debian:wheezy
MAINTAINER Marcus Stong, marcus@andyet.net
RUN apt-get update && apt-get install -y curl
CMD echo "Etcd Version: $(curl http://${COREOS_PRIVATE_IPV4}:4001/version)"

This Dockerfile returns the CoreOS Etcd version to confirm connectivity. In your real Docker containers, your applications can access Etcd by using the environment variable $COREOS_PRIVATE_IPV4 in the connection string.

Don't forget that if you're running a command in Docker as the core user in an ssh session, you have to run source /etc/environment first.

What About iptables?

If you're using iptables in CoreOS (hopefully you are) you'll need to include some additional rules. Assuming a default DROP policy on the INPUT, OUTPUT, and FORWARD tables, those rules would look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
#!/bin/bash
iptables -A FORWARD -i docker0 -o eth1 -j ACCEPT
iptables -A FORWARD -o docker0 -i eth1 -j ACCEPT

iptables -A INPUT  -i docker0 -p tcp --dport 4001 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o docker0 -p tcp --sport 4001 -m state --state ESTABLISHED -j ACCEPT

iptables -A OUTPUT -o eth1 -p tcp --dport 4001 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT  -i eth1 -p tcp --sport 4001 -m state --state ESTABLISHED -j ACCEPT

iptables -A INPUT  -i eth1 -p tcp --dport 4001 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o eth1 -p tcp --sport 4001 -m state --state ESTABLISHED -j ACCEPT

Conclusion

And there you go!

You should now be well on your way to using the Etcd goodness that makes CoreOS so attractive in the first place!

You might also enjoy reading: