How to Set Up Auto-Restart for Unhealthy Docker Containers

In this article we will see how to set up an Auto-Restart system for Docker containers that, for whatever reason, enter the unhealthy state. This is made possible by the willfarrell/autoheal container.

If a container becomes unhealthy we naturally want to restart it to get it working again, and ideally this would happen automatically. Unfortunately Docker does not offer this feature yet: it was proposed, but it has never been implemented.

Luckily there is the willfarrell/autoheal container, which regularly checks the status of your other containers and restarts the unhealthy ones.

What “unhealthy” means and why the restart policy is not enough

Many Docker images include a healthcheck: a command that Docker runs at regular intervals inside the container to verify that the application is really working (for example, that the web server is responding). Depending on the outcome of the command, the container can be in one of these states:

• starting: the container has just started and the healthcheck does not have a result yet;
• healthy: the healthcheck succeeds;
• unhealthy: the healthcheck has failed for the configured number of consecutive times (retries).

Docker’s restart policy (restart: always or unless-stopped) only restarts a container when its main process exits. An unhealthy container, on the other hand, still has its process running (perhaps stuck or no longer able to respond), so Docker leaves it as it is. This is where autoheal comes in.

We can check the health status of our containers with the command:

$ docker ps --format "table {{.Names}}\t{{.Status}}"

which will show, for example, Up 2 hours (healthy) or Up 5 minutes (unhealthy). To see only the unhealthy containers:

$ docker ps --filter health=unhealthy

Important: autoheal only acts on containers that have a healthcheck. A container without a healthcheck will never be unhealthy and will therefore never be restarted. In the “Testing” section below we will see how to add one in the docker-compose.yml.

There are two monitoring modes:

• monitor all containers and restart the ones in the unhealthy state;
• monitor and restart only a specific set of containers.

Below we will look at both methods.

A word on security

The willfarrell/autoheal image requires the container to have access to the host’s Docker socket: /var/run/docker.sock.

Mounting /var/run/docker.sock inside a container effectively gives the container, and everything running inside it, root privileges on the underlying host, since it becomes possible to do anything that a root user or a member of the docker group can do.

For example, someone could create a container, mount the host’s /etc folder, modify its configuration and open an attack vector to take control of the host.

The image has been around since 2017, it is very widely used, the source code is available for public inspection and many people run it (myself included), so the risk of malicious behaviour by the author is, in my opinion, quite low. However, if we decide to give this container access to the host’s /var/run/docker.sock, we do so at our own risk.

Monitoring and restarting all containers

Let’s create a dedicated folder and, inside it, the docker-compose.yml file:

$ mkdir ~/autoheal
$ cd ~/autoheal
$ nano docker-compose.yml

and paste the following:

services:
  autoheal:
    image: willfarrell/autoheal:latest
    tty: true
    container_name: autoheal
    restart: always
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

If you already have a docker-compose.yml file with other services, you can instead add the autoheal: block at the end of the services: section of that same file.

To explain what is happening in this configuration:

• we are creating a container called autoheal, from the willfarrell/autoheal:latest image;
• restart: always makes autoheal itself restart in case of problems and every time the host reboots;
• AUTOHEAL_CONTAINER_LABEL=all tells autoheal to monitor all containers;
• we mount the Docker socket /var/run/docker.sock, so that autoheal can control Docker (and restart containers).

Once the file is saved (CTRL+X, Y and ENTER), run the following command in the same folder:

$ docker compose up -d

to download the image and start the autoheal container. The container does not log anything as long as everything is fine, so we can check that it is running with the docker ps command, looking for the autoheal container: it should have a status of Up 20 seconds (healthy) or similar.

When it restarts an unhealthy container, autoheal writes it to its own log (which you can view with docker logs autoheal), for example:

15-05-2023 19:23:17 Container /adsbx (51bc3e3511f5) found to be unhealthy - Restarting container now with 10s timeout

Monitoring and restarting only a specific set of containers

To tell autoheal which containers to monitor, we need to apply a label to them. In this case the autoheal container must be configured without the AUTOHEAL_CONTAINER_LABEL=all variable: by default it only monitors containers with the autoheal=true label.

Labelling existing containers

To have autoheal monitor a container, add the following directive to the service definition in its docker-compose.yml file:

labels:
  - autoheal=true

The updated service may then look like this (note the added labels: section):

services:
  web:
    image: nginx:alpine
    container_name: web
    restart: always
    labels:
      - autoheal=true
    ports:
      - 8080:80
    tmpfs:
      - /run/test
    healthcheck:
      test: ["CMD-SHELL", "wget -q --spider http://localhost/ && test ! -f /run/test/fail"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s

Repeat the operation for all the containers you want autoheal to monitor, then apply the changes, in each application’s folder, with:

$ docker compose up -d

Docker Compose will notice the change and recreate the containers to which the label was added, leaving the others untouched. Nice, isn’t it?

Deploying the autoheal container

As in the previous case, create the ~/autoheal/docker-compose.yml file (or add the service to an existing file), this time without the environment variable:

services:
  autoheal:
    image: willfarrell/autoheal:latest
    tty: true
    container_name: autoheal
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Start the container with docker compose up -d and check with docker ps that it is running, exactly as in the previous method.

If we prefer to use a label name other than autoheal, we can set it with the AUTOHEAL_CONTAINER_LABEL variable (for example AUTOHEAL_CONTAINER_LABEL=restart-me): containers will then need the restart-me=true label.

Installing with a single command (docker run)

If we do not use Docker Compose, we can also start autoheal directly from the command line. To monitor all containers:

$ docker run -d \
    --name autoheal \
    --restart=always \
    -e AUTOHEAL_CONTAINER_LABEL=all \
    -v /var/run/docker.sock:/var/run/docker.sock \
    willfarrell/autoheal

To monitor only containers with the autoheal=true label, simply omit the -e AUTOHEAL_CONTAINER_LABEL=all line.

Configuration variables

The behaviour of autoheal can be customised with the following environment variables:

Variable	Default	Description
AUTOHEAL_CONTAINER_LABEL	autoheal	Name of the label used to select the containers to monitor. With the value all every container is monitored.
AUTOHEAL_INTERVAL	5	How often, in seconds, autoheal checks the status of the containers.
AUTOHEAL_START_PERIOD	0	Seconds to wait when autoheal starts before beginning the checks.
AUTOHEAL_DEFAULT_STOP_TIMEOUT	10	Seconds the container is given to stop before being forcibly killed during the restart. It can be set per container with the autoheal.stop.timeout label.
CURL_TIMEOUT	30	Maximum time (in seconds) for calls to the Docker API.
WEBHOOK_URL	empty	If set, autoheal sends a notification to this address every time it restarts a container.

For example, to check the containers every 30 seconds and allow 20 seconds for them to stop:

    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
      - AUTOHEAL_INTERVAL=30
      - AUTOHEAL_DEFAULT_STOP_TIMEOUT=20

Testing: checking that the auto-restart works

Before trusting the system, it is a good idea to verify that it really works. Let’s use the web container from the previous example (with the autoheal=true label). Its healthcheck does not just check that the web server responds: it also verifies that no file called fail exists in the /run/test folder. This folder is mounted as tmpfs (that is, in memory), so it is emptied every time the container restarts. It lets us “break” the container on demand, without touching its files.

Let’s create the fail file inside the container:

$ docker exec web touch /run/test/fail

After a few check cycles (in our example, 3 consecutive failures 30 seconds apart, so about a minute and a half) the container will become unhealthy. We can follow the progress with:

$ watch -n 5 'docker ps --format "table {{.Names}}\t{{.Status}}"'

As soon as the container turns unhealthy, autoheal restarts it: looking at the log with docker logs autoheal we will find a message similar to the one seen earlier. Since the /run/test folder is in memory, the fail file disappears with the restart, and after the start_period the container goes back to healthy.

Note: the restart performed by autoheal is equivalent to a docker restart, which does not recreate the container. Changes written to the container’s file system (for example a deleted file) therefore remain even after the restart, and if the problem depends on those changes the container will keep becoming unhealthy and will be restarted repeatedly. In that case you need to recreate it with docker compose up -d --force-recreate.

Common problems

• The container is never restarted: check that it has a healthcheck (docker inspect --format '{{json .State.Health}}' container_name must return something other than null) and that it is monitored, that is, it has the autoheal=true label or autoheal is configured with AUTOHEAL_CONTAINER_LABEL=all.
• The container keeps being restarted: the healthcheck is probably too strict, or the application takes longer than expected to start. Increase start_period, interval or retries in the service’s healthcheck.
• Docker socket access errors: check that the /var/run/docker.sock:/var/run/docker.sock volume is present and that the socket path on the host is correct.

Conclusion

With one extra small container we have made our Docker services able to “heal themselves”: when an application stops responding, autoheal restarts it without any intervention from us. I recommend combining this with a good backup of your Docker volumes, to stay safe even in the most serious cases.

If you have any problems with the configuration, feel free to leave a comment below.

Have fun!