14 May

Manage the Docker Swarm nodes – new Ansible 2.8 modules developed by me!

Ansible 2..8 introduces huge update for Docker Swarm modules

There are several good things about open source projects. But there is one which makes the good projects grow quickly – if you miss a feature and you have some programming skills you can write your extension. Then you can propose the change to get merged into the official repository. This is how my journey as a community Ansible developer started.

Lately, I worked more and more with Docker Swarm clusters and wanted to automate some tasks. Unfortunately, with the Ansible 2.7 there were only three modules available – thedocker_swarm, docker_swarm_serviceanddocker_secret. The tasks I wanted to automate required features as reading the Swarm configuration (this was partially covered by the docker_swarm), node status and change the node configuration. I had three options – use the shell module and execute the CLI command locally on the remote host, wait for someone to provide the module or write the module by myself.

I may not be a professional developer, but I had a great teacher when I started coding in the last class of primary school and some experience in programming. You can’t really be an engineer if you cannot write scripts or simpler apps. I think it is not a surprise I decided to test myself in professional project ūüėõ

In the previous post, I presented the docker_swarm_facts module. I am a co-author there. Let me now show you the Swarm Node related modules fully developed and now maintained by me. All of them will be available in Ansible 2.8 release in next few weeks, but you can test it using the devel branch code on GitHub!

Tell me about my Docker Swarm node…

Before you start changing the configuration of any device, software etc. you really should get and store some information about its current state. To read the essentials about the Docker Swarm node you need to use the docker_node_facts module. The module inherits all default docker modules options like the TLS configuration. You can execute it locally using the Docker socket or remotely using the Docker API. Remember that Docker by default use an only local socket so you need to reconfigure your daemon to support remote management. And one of the most important requirements – you need to run the module on Docker Swarm Manager node.

    - name: Get the swarm node facts
        self: yes
        docker_host: "tcp://"
        tls: true

By default, Ansible will open an SSH connection to the remote host and execute the module there connecting to local Docker socket. The host is usually an inventory entry. Using the API requires providing the URL of the management interface as the docker_host parameter. You can execute the module still on the remote host (the same as in docker_host or different), but usually, in such case, the playbook is set with the parameter connection: local and run on the localhost. Understanding this difference is crucial for running any Swarm module and avoid problems.

You also need to tell the module information on which Swarm node you want the manager to return. The default behavior is returning facts about all Swarm nodes. If you know the names of the nodes you are interested in you can provide them as a list in the name parameter. Setting the self: yes option will tell the module you want facts only about the node module communicates with.

Module output matches the output of the docker node inspect CLI command. The nodes key in the returned structure contains an array of dictionaries where each element matches the CLI command. Its structure may vary depending on the version of Docker daemon and Docker API – you need to check the documentation for details.

Changing the node configuration

The docker_node module allows changing some parameters of the Swarm node configuration like the node role or the availability. The first parameter defines if the node is a manager or a worker. Initially, you must define the role when you add a node to the cluster. The availability defines how the swarm cluster will use the node for work distribution. Three allowed states are active (accepting new containers), pause (operating the existing containers but not accepting new ones) or drain (not accepting new containers and existing will restart on another node). The last state is useful during upgrades or when you want to remove a node from the cluster

The docker_node module also allows changing the labels assigned to the Swarm node. Labels are usefull for management and automated work distribution. Labels are the key-value pairs. The default module behavior is merging the labels provided in playbook with those already assigned to node and update the value of a label if it is already defined.

- name: Merge node labels and new labels
    hostname: mynode
      key: value

To override default behavior and replace the labels with new ones you must set labels_state: replace. To remove all assigned labels you also have to use this option, but without providing any new labels.

- name: Remove all labels assigned to node
    hostname: mynode
    labels_state: replace

If you want to remove specified labels you need to provide the list of label keys in parameter labels_to_remove.

11 Mar

Using Docker Swarm? You gonna love Ansible 2.8!

Ansible 2..8 introduces huge update for Docker Swarm modules

The Ansible is kind of an icon of automation platform. Owned by RedHat but available as a free product on the public license. Developed both by RedHat employees and the community. Docker itself is an icon of containerization. If you use containers you know that automation is the key to simplify management of dockerized infrastructure. In Ansible 2.x many modules covering the docker operations has been introduced, but Docker Swarm was not really covered. However, it is going to change soon! There is a huge update coming with Ansible 2.8 release!

Lately, I started missing some features in Ansible that will allow me to perform some operations on Docker Swarm clusters. And I try to avoid using the command or shell modules as much as possible – running CLI commands on a remote host is like asking for troubles. I decided to fill this gap by myself and as a result, I can say I am now the Ansible community developer, author, and maintainer of a few modules – docker_swarm_facts, docker_node_facts, docker_host_facts, docker_node; and co-author of docker_swarm and the ansible.docker.swarm library.

In next posts, I wanna show you how those modules work. However, if you are looking forward using them I strongly advise you to give them a try now and report any bugs you find so we can fix them before Ansible 2.8 release. You can find them in the devel branch of Ansible repository.

Read More
17 Jan

Accessing host socket when using namespaces for Docker isolation

Socat will connect two sockets together

Nothing is secure by default and Docker is no exception. One of the recommended change to improve Docker security is isolation of the containers in user namespace which was introduced in Docker Engine 1.10. The namespaces makes the process run on the host thinks that it has its own access to some global resources like the PIDs. User namespaces provides the mechanism of remapping container resources to host resources limiting container access to the host system.

In some cases you may need to access the host resource from the container like the Docker own socket. It is required if you run Docker inside of the Docker container or you deploy a tool that will manage your Docker hosts or Docker Swarm cluster. The problem is the socket on the hosts is owned by the root, while the root PID from inside of your container is remapped to non-root PID on the host. It make the host socket inaccessible for the processes inside the container. But there is an easy workaround for this problem

Read More
04 Jan

Observium pollers vertical scaling

Observium in containers

There are several free monitoring tools available for commercial or non-commercial use. I decided to use Observium to monitor my home and remote labs. I had no specific requirements except that it had to have an option to run in Docker containers on RaspberryPi (ARMv6/8 CPUs). The downside of all monitoring solutions are the resource requirements. There are three resources that matter – storage, memory, and CPUs. Vertical scaling of Observium pollers and other processes is one of the solutions if your hardware resources are limited.

The vertical scaling means adding more workers doing the same process in parallel, but tasks assigned to each process should not overlap. In my case, I wanted to spread the polling across my RaspberryPi cluster. Polling process can consume the CPU and make RaspberriPi unresponsible. In worst scenario, if you have many devices or you are poling lots of data from your devices the polling processes may not finish their work within 5 minutes (this is how often devices have to be queried) so you will miss some data. You may of course tune number of threads the polling process start but any platform has it resources limitations you cannot bypass.  

In this article I will present my solution based on Docker Swarm cluster. It has some limitations and downsides but may work for many people, not only on RaspberyPi.

Read More
11 Dec

Upgrading the VMware Harbor

VMware Harbor is a docker images registry. You can use it instead of docer registry from official repository.

I lately decided to upgrade my local Docker registry installation. I use VMware Harbor as a Docker registry – In my opinion, it is much better and easier than official registry software. Recently I upgraded it from version 1.5.1 to 1.6.2.

VMware Harbor runs in containers which simplifies managing the software, but the upgrade is not straightforward as you may think. Most significant change is database consolidation. Instead of separate databases for Harbor, Clair and Notary version 1.6.0 introduces a single database engine for all components – the PostgreSQL. The 1.5.0 Harbor uses the MySQL, while Clair already uses the PostgreSQL. The developers prepared a dedicated container with migration engine that performs all the work. However, I found upgrade documentation missing the crucial explanation of steps and commands which may lead to loss of your data. I will try to cover my findings in this post.

Read More
24 Sep

If you build containers Alpine Linux is your friend

This post is related to Docker and automation

Every container image must start from a parent image or base image (the scratch). The parent image is the image you base your image on. The base image is like a completely empty container you need to fill with content. But in most cases, you will use another image as a parent, and you want it to be as minimal as possible. The Alpine Linux is your friend – remember this name and use it as much as possible.

Read More
20 Aug

Run Jenkins in the ‚Äčcontainer

Jenkins and Docker - flexible environment

The more you work with automation, the more you will like the containers. They fit and scale correctly in CI/CD model and can be easily managed. The whole infrastructure for automation should be flexible, easy to maintain and extendable – containers fit perfectly into this model. So why not start from putting Jenkins in the container?

Read More
05 Jul

Repository with Docker templates

Don't be afraid‚Äč of the source code

Everyone who worked on any project knows that sites like GitHub, GitLab or StackOverflow are full of open-source examples, functions, issue solutions and so on. I used them as well. It is not obligatory, but I feel that it is fair if I share something I create that may be useful for others. After all, I know what I was looking for in the past. No matter if we are creating Python script, the Java application or Docker templates.

I decided to create a small project called the DockerTemplates. The repository contains at the moment docker-compose files for various products I set up using Docker containers. I prefer using the docker-compose for everything even project consists only of one container. I find it the easiest way to maintain a consistent approach across multiple systems or projects. You will find the docker-compose configuration files in folders named after the application. In each folder, there is a separate documentation of what are the configuration details implemented in each file.

The initial release of this repository contains configuration files for following applications:

  • AWX
  • Jenkins
  • GitLab-CE

The repository will accept pull requests for any registered GitLab user so it is easy to participate and put your configuration. Please check requirements described in CONTRIBUTING.md guideline. You can also email files directly to me or leave a comment.

07 May

Why I use VMware Photon OS for Docker

Photon OS Project provides lightweight container-oriented operating system you can run as VM

One of my readers asked me what platform I use for my docker containers in the lab. He assumed it is one of the public cloud platform providing such service. This is not true. I run few containers in public cloud, but this is not cost-optimal for lab tests. Especially in the way I sometimes work, when there might be few hours gap between the tests. I run all the containers locally on my own infrastructure which does not consist of few racks in mine basement. Actually, it is very small, so it has to be resource-optimal. I decided that for now, VMware Photon OS Project is the best solution for me.

Read More

26 Apr

Container names and docker-compose

This post is related to Docker and automation

You learn new things every day. Quite often by accident because something is not working as you expect or it just stopped working. This is one of those situations where I learned to be careful how to use container names using docker-compose and how containers refer to each other names. This is very important if they share the same network.

To perform some tests I created a lab where I run Ansible AWX (the free Ansible Tower version) and Netbox IPAM. I decided to use docker-compose because it gave me an easy way to manage the lab. By default, both AWX and Netbox is put in dedicated Network just for the containers of each installation. This means the isolation Рthe desired way you want to run containers. But in my lab, AWX need to talk to Netbox API to pull inventory. Because it is my lab I did the simplest thing.

Read More