CoreOS: Etcd in Docker containers

December 17, 2014Marcus Stong

At &yet we have become big fans of CoreOS and Docker, and we've recently moved a bunch of our Node applications there.

We wanted to share a little tidbit for something that seems like it should be easy, but ended up being a minor stumbling block: making CoreOS's Etcd cluster available in Docker.

Here's a little background for those not up to speed on CoreOS or Etcd. CoreOS bills itself as "Linux for Massive Server Deployments." CoreOS is usually deployed with Etcd, "a highly-available key value store for shared configuration and service discovery," giving servers in a cluster the ability to share configuration data. We have been using Etcd and Confd for some time, so we naturally thought: "wouldn't it be cool to configure applications and services within Docker from Etcd?"

This can be easily accomplished by passing an environment variable to Docker at runtime and tweaking a few iptables rules. CoreOS makes the Etcd HTTP API available on its private IP address at port 4001 by default. CoreOS also sets an environment variable called $COREOS_PRIVATE_IPV4 in /etc/hosts with said IP. Fortunately, Docker is networked to the host's private IP, so this is the easiest point of access for Etcd.

Here's a sample unit file for Fleet and Dockerfile with the needed configuration to bring it all together:

Unit File

[Unit]
Description=Node App
Requires=docker.service

[Service]
EnvironmentFile=/etc/environment
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill node-app
ExecStartPre=-/usr/bin/docker rm node-app
ExecStartPre=/usr/bin/docker pull {{docker_registry}}/node-app
ExecStart=/usr/bin/docker run --rm=true --name=node-app -P -e COREOS_PRIVATE_IPV4=${COREOS_PRIVATE_IPV4} registry/node-app
ExecStop=/usr/bin/docker stop node-app

There are two relevant parts of the unit file to make it all work. First, EnvironmentFile=/etc/environment must be set, otherwise ${COREOS_PRIVATE_IPV4} will be empty. The second is setting -e COREOS_PRIVATE_IPV4=${COREOS_PRIVATE_IPV4} so your container also has access to the private IP.

Dockerfile

FROM debian:wheezy
MAINTAINER Marcus Stong, marcus@andyet.net
RUN apt-get update && apt-get install -y curl
CMD echo "Etcd Version: $(curl http://${COREOS_PRIVATE_IPV4}:4001/version)"

This Dockerfile returns the CoreOS Etcd version to confirm connectivity. In your real Docker containers, your applications can access Etcd by using the environment variable $COREOS_PRIVATE_IPV4 in the connection string.

Don't forget that if you're running a command in Docker as the core user in an ssh session, you have to run source /etc/environment first.

What About iptables?

If you're using iptables in CoreOS (hopefully you are) you'll need to include some additional rules. Assuming a default DROP policy on the INPUT, OUTPUT, and FORWARD tables, those rules would look like:

#!/bin/bash
iptables -A FORWARD -i docker0 -o eth1 -j ACCEPT
iptables -A FORWARD -o docker0 -i eth1 -j ACCEPT

iptables -A INPUT  -i docker0 -p tcp --dport 4001 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o docker0 -p tcp --sport 4001 -m state --state ESTABLISHED -j ACCEPT

iptables -A OUTPUT -o eth1 -p tcp --dport 4001 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT  -i eth1 -p tcp --sport 4001 -m state --state ESTABLISHED -j ACCEPT

iptables -A INPUT  -i eth1 -p tcp --dport 4001 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o eth1 -p tcp --sport 4001 -m state --state ESTABLISHED -j ACCEPT

Conclusion

And there you go!

You should now be well on your way to using the Etcd goodness that makes CoreOS so attractive in the first place!

&yet

The Blog