patg.net

home

CoreOS DNA on Debian

27 Jan 2015

CoreOS DNA on Debian

For a month now, I've been meaning to post about the great work my team, the Advanced Technology Group at HP, did with regard to Docker and a proof-of-concept that we embarked on. Our project was named "Inserting CoreOS DNA on Debian for Creating Docker Clusters".

The goal was to create a proof of concept (POC) to meld the best parts and be usable for experimentation. We examined the criteria and choices for creating the clustered Docker POC and demonstrated its use.

Why Debian?

HP Helion runs on Debian and we wanted to see how feasible this was as well as introduce this interesting combination of components to others.

What is CoreOS?

The strategies and architectures that influence CoreOS allow companies like Google, Facebook and Twitter to run their services at scale with high resilience.

CoreOS is a minimalist Linux system based on Chrome OS and uses 40% less RAM on boot than the average Linux installations. It has been rearchitected to provide features to run modern infrastructure stacks by facilitating the building of customized platforms depending on the requirements of a given invidual or organization.

As mentioned on CoreOS's website, its development and architecture is intended to allow companies like Google, Facebook, and Twitter to run their services at scale and with high resilience.

CoreOS includes the necessary components to run Docker clustered across machines. The components are:

Implementation

Using HP’s Public Cloud, built from OpenStack, we constructed a compute (VM) image utilizing the aforementioned components and orchestrated with Ansible on top of a stock (or default) Debian “Testing” installation.

Using the built image, we launched a given number, in our case, 5 VMs to demonstrate the usefulness of fleet with a sample ELKstack application. The ELKStack sample app was a modified version of the one given by Marcel DeGraaf in his excellent blog post. In addition to his great article, he also provides example code. We used this example, slightly modified, to demonstrate our proof of concept worked in a practical way supporting a useful app as well as showing how fleet intelligently scheduled docker containers across machines in the cluster.

The Sample Application

The sample application consisted of the following containers and numbers when run:

Systemd magic

Each of these containers is specified within a systemd service file that fleet utilizes when interacting with systemd on each node of the cluster. Systemd provides a means to ensure that a service (corresponds to container) has a distinct name and that there cannot be conflicts. Also systemd provides a means to run certain things before and after a service is started. Some of these are:

This is an extremely useful functionality systemd provides that makes applications like this possible!

Launch order

For all of these containers except the Kibana/Elasticsearch container, they each need to have the ability once running to know the IP address of the other containers. Logstash needs to know the IP of Kibana/Elasticsearch in order for forward events to, the Sinatra web app containers need to know the IP address of the Logstash container for each logstash-agent to forward events to the Logstash central logging agent, the Nginx container needs to know which Kibana web application containers there are to proxy to. The test script container needs to know what the IP is of Nginx.

This discovery capability is provided by the very thing that makes Fleet able to do its job: etcd

These containers are luanched using the `fleetctl utility (once the service file is loaded) in the given order listed above. When each container is launched, the service file for each specifies a step to be run immediately after the service is launch which in the case of these services, the IP address of each is stored in etcd to be in turn used by the next container launched.

For instance, when the Elasticsearch container is launched, it registers its IP address as elasticsearch/host in etcd. Each of these containers is "baked" with another great utility that is luanched with a simple boot.sh script: confd

Confd: another magic ingredient

confd on each container is configured to generated the necessary configuration files using a confd template that interpolates the variable provided from etcd.

This can be explained better this way:

The one thing this sample app for our POC made clear to me: the very thing that makes the docker clustering work can in turn provide a great means of providing flexible discovery for a given application. The one thing that comes in mind for me that I've had to make work with numerous tricks through the years using Chef, Saltstack, and Ansible is Galera clustering.

Summary

There is so much about the proof-of-concept that we worked on that I want to write about and will have subsequent posts about a number of things such as other Docker clustering and scheduling projects, the ansible work that was done, the work we did to make this work on HP Cloud considering private and public IPv4 addresses, as well as more details on how to use confd which is a tool that I really had fun aquainting myself with.

Stay tuned!

comments powered by Disqus