Upgrading to Traefik 2 with Docker

December 13, 2019

I’m running some web services for personal use. These services are mostly running from containers with a reverse proxy to expose them to the web. I leave this task to Traefik.

Before that I used a popular triad of nginx, docker-gen and letsencrypt-nginx-proxy-companion. This for the most part worked great but three services for one task also meant three sources for possible errors. Around this time last year I learned about Traefik with its single binary approach and general small footprint. After following a guide by Keith Thompson I jumped ships.

This fall containous the company behind Traefik released version 2.0. This is a major release including cool stuff like reusable middlewares, a new fun web dashboard and advanced stuff for production deployments like canary deployments. This changes come with a trade off. The new version has lots of breaking changes because of that I had to update my deployment and understand the new paradigms introduced. In this post I want to help share my findings and show a path to upgrade from 1.7 to 2. So without further ado, let’s jump right in.

Step 0 - Our initial Traefik v1.7 setup

Before we go further, we are setting up a small Docker based environment in Traefik 1.7 that we are going to upgrade. For that I’ll assume that you already have docker and docker-compose installed and know your way around these tools.

We are exposing two Docker containers and Traefik’s dashboard. The setup is quite simple and can be described as follows: Of course you need to use your own subdomains

Folder structure:

├── acme.json # stores SSL certs and keys
├── docker-compose.yml # defines containers
├── htdocs
│   └── index.html # super awesome website
└── traefik.toml # Traefik`s configuration:
                 # defines entrypoints, enables dashboard,
                 # ACME, Docker provider and access logging

I created a repository at GitHub that follows our upgrade steps. You can take a look at the initial situation in this commit.

From Frontends and Backends to Routers, Services and Middlewares

Before we upgrade this setup we need to understand some key differences between Traefik v1.7 and v2. The notion of “Frontends” and “Backends” is gone, in the new version the concepts of “Routers”, “Middlewares” and “Services” were introduced. Especially against the backdrop of our planned upgrade, these concepts can be considered as follows:

• Frontends become Routers
• Backends become Services
• Things like frontend.redirect or frontend.headers become middlewares

The great part about middlewares is that they can be reused. For instance a scheme redirection from http to https can be defined once and used again all over the setup by just adding it to a router.

Please check out the documentation for more background on these new concepts.

Step 1 - Upgrading the Traefik dashboard

We will start of by focusing on the basic setup and enabling dashboard. Begin by stopping your 1.7 setup with docker-compose stop && docker-compose rm -f after that comment out the services whoami and httpd, they’ll be enabled again later on. Next we’ll replace the tag of the Traefik image with v2.1.

Like before Traefik uses a static and a dynamic configuration. The first are set during startup. The later are fully dynamic and can change while Traefik is running.

In v2 both can’t be mixed that’s why we are creating two new configuration files. The new version also introduced YAML support, since we are already using it for docker-compose I suggest we switch from TOML to YAML. Of course this is a purely subjective decision on my part.

$ touch dynamic_conf.yml traefik.yml

Both files need to be mounted in our docker-compose.yml file like this:

volumes:
      - ./traefik.yml:/traefik.yml
      - ./dynamic_conf.yml:/dynamic_conf.yml

Static configuration (Entrypoints, ACME, Providers…)

Now we can start reworking our static configuration traefik.yml. I’m following the structure of the old TOML file from top to bottom.

Entrypoints

First lets refactor the Entrypoints to YAML. In the same step we’re also removing the basic authentication for the dashboard/API and the https redirect. Don’t worry they’ll be added again as middlewares 🚀

entryPoints:
  dashboard:
    address: :8080
  http:
    address: :80
  https:
    address: :443

Dashboard and API

Next up the Dashboard/API. We just tell Traefik to enable both like this:

api:
  dashboard: true

The Entrypoint definition will be part of our dynamic configuration as part of a router. Remember in v2 static and dynamic configuration don’t mix!

ACME (Let’s Encrypt)

First up: In v2 the format of the acme.json changed. This file needs to be converted from v1 to v2. For this task containous released the traefik-migration-tool. Run it from your working directory:

$ docker run --rm -v ${PWD}:/data containous/traefik-migration-tool \
    acme -i /data/acme.json \
    -o /data/acme-new.json
$ sudo chmod 600 acme-new.json
$ sudo mv acme-new.json acme.json

ACME now is a certificate resolver that needs to be defined:

certificatesResolvers:
  myhttpchallenge:
    acme:
      httpChallenge:
        # used during the challenge
        entryPoint: http
      email: hi+letsencrypt@moritzvd.com
      storage: acme.json
      # Use Let's Encrypt Staging CA when testing!
      # caServer: https://acme-staging-v02.api.letsencrypt.org/directory

Providers

The documentation tells us:

Configuration discovery in Traefik is achieved through Providers.

For us that means we need two providers docker and file the latter for our dynamic_conf.yml that we’ll be adding in a couple of minutes.

providers:
  docker:
    exposedByDefault: false
    network: web
  file:
    filename: dynamic_conf.yml
    watch: true

Access Logging and Traefik Debugging

We want to keep logging access and also increase the verbosity of Traefik. That’s why we just add:

log:
  level: INFO
accessLog: {}

Our static configuration is now finished.

Dynamic Configuration for Dashboard and API

Now that we have defined the static configuration we need to work on our dynamic routers and middlewares. Remember they have the job of exposing our services and can change while Traefik is running.

Like on 1.7 we want our dashboard to be available on :8080 and https. For that we need two Frontend routers that expose the service api@internal. These routers also need a use a basicAuth middleware so that users are asked for our super secure passw0rd.

Check create_docker_compose_basic_auth_string_for_traefik.sh (my fork at gist) to generate you own authentication string.

We start by defining our basic authentication middleware. We use the same password that was used in the old traefik.toml.

In dynamic_conf.yml:

http:
  middlewares:
    dashboard-auth:
      basicAuth:
        users:
          # admin:passw0rd
          - "admin:$2y$05$kqK7GlhnGCnt/fYdrCL2AeZykK2T0cN4sBcCaRs61vMz7yMWaR9M."

Next we define our first router:

routers:
    my-api:
      entryPoints:
        - dashboard
      rule: "PathPrefix(`/dashboard`) || PathPrefix(`/api`)"
      service: api@internal
      middlewares:
        - dashboard-auth

What did we do here? This router gets activated when the Route '/dashboard' or '/api' gets requested (the given rule condition is fulfilled). It adds the dashboard-auth middleware and than forwards the request to the api@internal service. The router is only exposed on Port 8080 aka our dashboard entrypoint.

No we can run first tests on our setup. The Traefik API should reply with a list of our active routers:

$ docker-compose up -d
$ curl -s --user admin:passw0rd http://localhost:8080/api/http/routers | jq

[
  {
    "entryPoints": [
      "dashboard"
    ],
    "middlewares": [
      "dashboard-auth@file"
    ],
    "service": "api@internal",
    "rule": "PathPrefix(`/dashboard`) || PathPrefix(`/api`)",
    "status": "enabled",
    "using": [
      "dashboard"
    ],
    "name": "my-api@file",
    "provider": "file"
  }
]

Next we add a second router that is in charge of securely exposing the dashboard and API via https. It’s mostly the same but uses a more precise rule and our ACME HTTP Challenge from traefik.yml. The longer rule gets a higher priority by Traefik than the one defined for port 8080.

my-secure-api:
      entryPoints:
        - https
      # Activate this router if Client requests specific subdomain and '/dashboard' or '/api'
      rule: "Host(`dashboard.traefik.moritzvd.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      service: api@internal
      middlewares:
        - dashboard-auth
      tls:
        # Use ACME HTTP Challenge defined in 'traefik.yml' to get valid cert
        certResolver: myhttpchallenge

Now we can access the Dashboard securely from our Browser:

Finally we want to redirect from http:// to https://. This great article by Gérald Croë gave me the idea to just do this as a general catch-all router. Since at the end we all just want https everywhere right?

For this we first define a redirectScheme middleware:

redirect-to-https:
      redirectScheme:
        scheme: https
        permanent: true

Now we add our catch-all router:

https-redirect:
      entryPoints:
        - http
      # Activate this Router on any Host requested
      rule: "hostregexp(`{host:.+}`)"
      service: dummy
      middlewares:
        - redirect-to-https

Because Traefik always requires a service when defining routers in a configuration file we add a “dummy” service. This service will never be reached because the middleware will redirect them to the requested service on https.

The service of course needs to be defined:

services:
    dummy:
      loadBalancer:
        servers:
          - url: localhost

Now all clients get permanently redirected to the secure version:

$ curl -sI http://dashboard.traefik.moritzvd.com/api/http/routers
HTTP/1.1 308 Permanent Redirect
Location: https://dashboard.traefik.moritzvd.com/api/http/routers
Date: Tue, 03 Dec 2019 16:24:09 GMT
Content-Length: 18
Content-Type: text/plain; charset=utf-8

Step 2 - Upgrading the whoami service

Now that we have the nice dashboard and the practical API running we focus on our services. As before we directly setup the Traefik configuration via labels in our docker-compose.yml. We start of by upgrading the simple whoami service. The service is a “tiny Go web server that prints OS information and HTTP requests” (containous/whoami).

First we uncomment the services definition that we disabled before.

Our old Traefik 1.7 service definition looks like this:

whoami:
    image: containous/whoami
    container_name: whoami
    restart: unless-stopped
    networks:
      - web
    labels:
      - "traefik.enable=true"
      - "traefik.backend=whoami"
      - "traefik.frontend.rule=Host:whoami.traefik.moritzvd.com"
      - "traefik.docker.network=web"

The former configuration tells us the following: We need one router that activates when the whoami.traefik.moritzvd.com is requested. We name this router whoami-secure as the name suggests we only want it to be accessible via https.

We define the router and the rule and the entrypoint in two label lines:

# The router is named inline
      - "traefik.http.routers.whoami-secure.entrypoints=https"
      - "traefik.http.routers.whoami-secure.rule=Host(`whoami.traefik.moritzvd.com`)"
      - "traefik.http.routers.whoami-secure.tls.certresolver=myhttpchallenge"

Because we defined a global catch-all redirect router the http to https redirection is already taken care of. Our finished service definition looks like this:

whoami:
    image: containous/whoami
    container_name: whoami
    restart: unless-stopped
    networks:
      - web
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami-secure.entrypoints=https"
      - "traefik.http.routers.whoami-secure.rule=Host(`whoami.traefik.moritzvd.com`)"
      - "traefik.http.routers.whoami-secure.tls.certresolver=myhttpchallenge"

We can now bring up the service by closing the file. Docker compose will pick up on the new service.

$ docker-compose up -d
traefik is up-to-date
Creating whoami ... done

Traefik will generate a new certificate and key, and will ask Let’s Encrypt to give us a signed certificate. This normally happens within a couple of seconds. After that we can access the service:

$ curl -s 'https://whoami.traefik.moritzvd.com'
Hostname: edf3d71d3412
IP: 127.0.0.1
IP: 172.18.0.3
RemoteAddr: 172.18.0.2:39308
GET / HTTP/1.1
Host: whoami.traefik.moritzvd.com
User-Agent: curl/7.58.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: [...]
X-Forwarded-Host: whoami.traefik.moritzvd.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 248ac49c7b90
X-Real-Ip: [...]

Step 3 - Upgrading the httpd service

Now we can upgrade our final service. For the sake of this guide we want to expose a good old Apache httpd. We also want to add some security headers like STS and CORS Headers. These become part of a new middleware that we call in our router.

In order to upgrade, we first uncomment the old Traefik 1.7 style definition. Afterward we do the following modifications:

Delete the lines:

- "traefik.backend=httpd" # The new router is named inline
    - "traefik.docker.network=web" # already defined this as default in `traefik.yml`
    - "traefik.port=80" # We are using the standard port exposed by httpd

Than we define our new router. We’ll call it httpd-secure and we want to expose it on our https entrypoint.

- "traefik.http.routers.httpd-secure.entrypoints=https"
    - "traefik.http.routers.httpd-secure.rule=Host(`traefik.moritzvd.com`)"
    - "traefik.http.routers.httpd-secure.tls.certresolver=myhttpchallenge" # Don't forget the certresolver

Now we need to rework our header definitions. Fortunately, the definitions have remained the same. But they now need to be defined as part of a headers middleware. We just need to Replace traefik.frontend.headers with traefik.http.middlewares.httpd-security.headers.

Our labels now look like this:

- "traefik.http.middlewares.httpd-security.headers.browserXSSFilter=true"
      - "traefik.http.middlewares.httpd-security.headers.customFrameOptionsValue=SAMEORIGIN"
      - "traefik.http.middlewares.httpd-security.headers.forceSTSHeader=true"
      - "traefik.http.middlewares.httpd-security.headers.frameDeny=true"
      - "traefik.http.middlewares.httpd-security.headers.SSLHost=traefik.moritzvd.com"
      - "traefik.http.middlewares.httpd-security.headers.SSLRedirect=true"
      - "traefik.http.middlewares.httpd-security.headers.STSIncludeSubdomains=true"
      - "traefik.http.middlewares.httpd-security.headers.STSSeconds=63072000"

By doing this we created a new middleware called httpd-security. Of course this middleware needs to be added to our httpd-secure router. This middleware could also be added to any other service that is in need of some nice security headers.

- "traefik.http.routers.httpd-secure.middlewares=httpd-security"

After firing docker-compose up -d all our services are now back again and served with Traefik 2.1. 🏁

I hope I could help in upgrading your setup from 1.7 to 2.1. We could go the extra mile and refactor everything even further by including all of the configuration in the docker-compose.yml file.

Credits for sources used in the banner: Background vector created by vectorpouch - www.freepik.com and the Traefik logo licensed under the Creative Commons 3.0 Attributions license.