Lab 10 - DevOps¶

This week's topic is a small introduction to Docker and DevOps union deployments wise. We will be looking into how to deploy Docker applications so that running them does not require you to change configuration files every time.

Please make sure you did the previous lab, especially editing the /etc/docker/daemon.json file.

This lab is composed of the following topics:

• Debugging containers
• Linking containers
• Properly publishing a service running in a container using a dynamic proxy
• Pushing a self built container to a registry

Debugging containers¶

Often enough, things do not work as you want them to. Either requests do not return what you would expect, the container itself seems to be missing data, some dependency you did not account for, etc.

One must know how to find debug information from containers for these reasons.

Logs¶

The problem is that even though the docker daemon writes logs:

• journalctl -r -u docker

You'll notice that these logs are only about the docker service itself. You care more about the container logs. These can be checked doing:

• docker logs <container ID|name>

So, for an example, doing docker logs <docker_lab_container_name> should return you this:

192.168.251.1 - - [11/May/2020:03:47:16 +0000] "GET / HTTP/1.1" 200 7234 "-" "curl/7.66.0" "-"
192.168.251.1 - - [11/May/2020:03:48:36 +0000] "GET / HTTP/1.1" 200 7234 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:75.0) Gecko/20100101 Firefox/75.0" "-"

Which are the same as the access logs for Apache. All the container errors get put here as well if the software and dockerfile is set up properly. You can also read the logs information from stopped (not deleted) containers, which is especially handy for finding why they stopped.

Internals¶

Logs are not always helpful. Sometimes you want to debug something inside the container or test network connectivity.

For this reason, you can (sometimes, when the creator of the Dockerfile has left appropriate binaries in) open a shell inside a container and use it as your own VM.

Try doing this: docker exec -ti docker_lab sh

• exec means to execute a command inside a container
• -t means to open a tty (shell), and -i makes it interactive so that you can type there
• docker_lab is the container name of the container we built an image for.
• sh is the command to be executed

If this works, it should drop you into a new shell. If you check the files and IP address, it is an entirely different machine. You can check the configuration used to run the container (cat /etc/passwd), install software (apk update && apk add vim), and ping the network to test networking.

Some things to remember:

• If you make changes to this container from the inside, then (unless the changes are made in a mounted directory) it will be there only for the lifetime of this particular container. The moment you delete this one, it's gone. For this reason, it's never a good practice to make changes inside the container, but always the Dockerfile.
• localhost now means the container itself.
• If you do curl localhost:5000, now you see the same information on port 5000 as you did on port 5005 before.
  • You may need to first install curl (apk add curl)
• Port 5005 gives an error because nothing is listening inside the container on port 5005.
• If you want to access your VM host, not the container, you need to use an IP address. If you check your container's IP address, it should be something like 192.168.67.X. You can access your VM on 192.168.67.1 (e.g. curl 192.168.67.1:5000).

Security scanning¶

One of the more significant issues with packaging software like containers is that it's challenging to keep track of what versions of software/operating system/tools you are running in each container.

It's even worse when trying to find container images. How can you make sure the image you want to use is safe?

Thankfully, nowadays, we have tools called Security Scanners. There's quite a few of them, but we will be using Trivy, which does not need any other dependencies to run.

To install Trivy on your VM, run the required commands from here: Trivy getting started

Scanning with Trivy is a relatively simple ordeal. All you need to do, is give it the name of either a local image, like:

trivy image registry.hpc.ut.ee/mirror/library/alpine

Or scan an image in a remote registry:

trivy image registry.hpc.ut.ee/mirror/library/python:3.4-alpine

As you can see, the Alpine image has no outstanding security issues, which is excellent. On the other hand, Python image is old enough to have quite a few.

these issues can be solved by either recreating the entire image or updating it when using it as a base image for projects.

Container optimization¶

In the last lab we touched on writing dockerfiles and how to build images from them. You can easily write a detailed dockerfile and publish it via docker build . All of the commands that you wrote will be turned into layers for the image and stored, increasing the image size. Managing container size and optimization is an important factor of DevOps. Well optimized containers do exactly what you need them to do and not a bit more. Optimization also potentially decreases the image size and complexity, making scaling, storing and readability a lot easier.

The dockerfile lines themselves give a good indication on what is being run, however tools can be used to see what is actually happening under the hood. The tool we are going to use today is called dive . You can install it on your system but it is not required to complete the lab as we will be using the Docker installation. This means that the executable command will be a little bit longer as well

Using dive is simple, just run: docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock registry.hpc.ut.ee/mirror/wagoodman/dive:latest <image name> . We will inspect the image that we created in the last lab, docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock registry.hpc.ut.ee/mirror/wagoodman/dive:latest docker_lab .

Linking containers¶

Sometimes, you want to hide your running service from the outside world for security reasons, but still allow it to be accessible by some containers.

A good example is a database. A database does not need to be accessible from the outside world, as it contains important information that is very easy to access - you only need to know the right username and password.

This is why Docker uses networks internally, which is one of the most complicated aspects of Docker. We will now build a new container analogous to the docker_lab container but with updated code. We will also deploy a mariadb container to store the access logs and motd messages of the applications.

As the centos user, create a directory to contain your files with which to build your container, and cd to that directory¶

mkdir -p ~/devops_lab/container
cd ~/devops_lab/container

Dockerfile:

FROM registry.hpc.ut.ee/mirror/library/alpine

RUN apk add --no-cache python3 py3-flask py3-sqlalchemy py3-sqlalchemy-utils
RUN apk add --no-cache py3-mysqlclient

COPY server.py /opt/server.py

EXPOSE 5000
CMD python3 /opt/server.py

server.py:

#!/bin/env python3

from flask import Flask, request, jsonify, redirect
import socket
from traceback import format_exc
import os
import time
from sqlalchemy import Table, Column, String, MetaData, Engine, Integer, create_engine
from sqlalchemy_utils import create_database, database_exists
from logging import Logger, getLogger

class MOTDStore():
    engine: Engine
    metadata: MetaData
    oauth_states: Table

    @classmethod
    def build_motd_store_table(cls, metadata: MetaData) -> Table:
        return Table(
            "motd",
            metadata,
            metadata,
            Column("id", Integer, primary_key=True), # Auto-increment should be default
            Column("date", String(20)),
            Column("message", String(200))
        )

    @classmethod
    def build_last_access_store_table(cls, metadata: MetaData) -> Table:
        return Table(
            "access_times",
            metadata,
            metadata,
            Column("id", Integer, primary_key=True), # Auto-increment should be default
            Column("date", String(20)),
            Column("ip", String(16))
        )

    def __init__(
        self,
        logger: Logger = getLogger(__name__)
    ):
        self._logger = logger
        db_password = os.environ.get("DATABASE_ROOT_PASSWORD")
        url = "mysql+mysqldb://root:%s@devops_lab_db:3306/motd_database"%(db_password)
        self.engine = create_engine(url)
        if not database_exists(url):
            create_database(url)
        self.metadata = MetaData()
        self.motd_store = self.build_motd_store_table(self.metadata)
        self.last_access_store = self.build_last_access_store_table(self.metadata)
        self.create_tables()

    def create_tables(self):
        self.metadata.create_all(self.engine)

    @property
    def logger(self) -> Logger:
        if self._logger is None:
            self._logger = logging.getLogger(__name__)
        return self._logger

    def add_motd(self, date: str, message: str) -> None:
        with self.engine.begin() as conn:
            conn.execute(
                self.motd_store.insert(),
                {"date": date, "message": message},
            )

    def log_access(self, date: str, ip: str) -> None:
        with self.engine.begin() as conn:
            conn.execute(
                self.last_access_store.insert(),
                {"date": date, "ip": ip},
            )

    def get_motd(self) -> str:
        with self.engine.begin() as conn:
            c = self.motd_store.c
            query = self.motd_store.select().order_by(c.id.desc())
            result = conn.execute(query).first() #Select only the last entry
            if result:
                return result[2]
            return "No message has been added yet. <a href='/new-motd'>Visit here</a> to add a new message of the day."

#https://stackoverflow.com/questions/4060221/how-to-reliably-open-a-file-in-the-same-directory-as-the-currently-running-scrip
__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))

app = Flask(__name__)

motd_store = MOTDStore()
def create_tables(self):
    self.metadata.create_all(self.engine)

@property
def logger(self) -> Logger:
    if self._logger is None:
        self._logger = logging.getLogger(__name__)
    return self._logger

def add_motd(self, date: str, message: str) -> None:
    with self.engine.begin() as conn:
        conn.execute(
            self.motd_store.insert(),
            {"date": date, "message": message},
        )

def log_access(self, date: str, ip: str) -> None:
    with self.engine.begin() as conn:
        conn.execute(
            self.last_access_store.insert(),
            {"date": date, "ip": ip},
        )

def get_motd(self) -> str:
    with self.engine.begin() as conn:
        c = self.motd_store.c
        query = self.motd_store.select().order_by(c.id.desc())
        result = conn.execute(query).first() #Select only the last entry
        if result:
            return result[2]
        return "No message has been added yet."

#https://stackoverflow.com/questions/4060221/how-to-reliably-open-a-file-in-the-same-directory-as-the-currently-running-scrip
__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))

app = Flask(__name__)

motd_store = MOTDStore()

def generate_time_string():
    return time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())

def log_access_to_db(ip):
    try:
        motd_store.log_access(date=generate_time_string(), ip=ip)
    except FileNotFoundError as e:
        app.logger.error(format_exc())

@app.route("/")
def hello():
    response = "Client IP: " + request.remote_addr + "\nHostname: " + socket.gethostname() + "\n"
    return time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())

@app.route("/motd")
def message_of_the_day():
    log_access_to_db(request.remote_addr)

    response = "<h1>The message of the day is:</h1><br>"

    try:
        response += motd_store.get_motd()
    except FileNotFoundError as e:
        response += "No message today"
        app.logger.error(format_exc())
    response += "<br><br><br><a href='/new-motd'>Visit here</a> to add a new message of the day."


    return response, 200

@app.route("/add-motd")
def add_motd():
    motd = request.form.get("motd") or request.args.get("motd")
    if motd:
        app.logger.info("Setting new motd:\n%s"%(motd))
        motd_store.add_motd(date=generate_time_string(), message=motd)
    return redirect("/motd", code=302)

@app.route("/new-motd")
def new_motd():
    form = """
<form action="/add-motd">
  <label for="motd">Set new message of the day:</label><br>
  <input type="text" id="motd" name="motd"><br>
  <input type="submit" value="Submit">
</form>
"""


    return form, 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Finally, build your container and name it devops_lab:

docker build -t devops_lab .

Run a mariadb container¶

But first, you need to create a directory you can mount to the container to keep your databases.

cd ~/devops_lab
mkdir db_data

Now you are ready to run your mariadb container:

docker run -d --name devops_lab_db -e MYSQL_ROOT_PASSWORD=<SOME PASSWORD FOR DB ROOT> -v $(pwd)/db_data:/var/lib/mysql registry.hpc.ut.ee/mirror/mariadb

Run the devops_lab container¶

docker run -d --name devops_lab -e DATABASE_ROOT_PASSWORD=<THE ROOT PASSWORD YOU SET WHEN RUNNING THE MARIADB CONTAINER> devops_lab

Create a docker network and make the mariadb container available only from this internal docker network.¶

First create a network for the containers of this lab and verify that the network was created

docker network create devops_lab --subnet 192.168.150.0/24
docker network ls

Then connect both the devops_lab and devops_lab_db containers to the network

docker network connect devops_lab devops_lab
docker network connect devops_lab devops_lab_db

Now try starting the devops_lab container again. This time it should start, verify it with docker ps

docker start devops_lab
docker ps

There are tools to make this easier for you, for an example, docker-compose, but we find that you cannot use tools properly unless you know what actually happens, when you do use those tools.

Properly publishing a container¶

Even though using the ports method for publishing things to the internet would work.. technically, then there are huge problems with that approach:

• You need to remember which service is on which port.
• You cannot scale services, as you cannot put multiple of them listening on the same port.
• If you use a service provider, then it is very often that only some ports from the internet are allowed. (e.g. in the public internet, only ports 80 and 443)
• Firewall and security configuration becomes complicated.
• You have no overview about how often or how your service is used unless the software inside your container provides that information. (it usually does not)

One of the solutions would be to do a localhost proxy like we did in the web lab or last lab. The problem with this is, that it would solve only points 2, 3 and 5. Thankfully, there are thought out solutions out there capable of proxying without using any docker ports.

One of these services is called Traefik. We will be setting up a Traefik proxy to a container without dabbling with the fancy buttons and dials Traefik has (automatic encryption, metrics, logging, complicated routing). This is left as homework, if interested.

First, create a directory and a file for traefik config

sudo mkdir /etc/traefik
sudo touch /etc/traefik/traefik.toml

After that populate the config file with the following:

[global]
  checkNewVersion = true
  sendAnonymousUsage = true

[entryPoints]
  [entryPoints.web]
    address = ":80"

[log]

[api]
  insecure = true
  dashboard = true

[ping]

[providers.docker]

Then run a traefik container with the config file and /var/run/docker.sock mounted to the container.

docker run --name traefik -d -p 50080:80 -v /var/run/docker.sock:/var/run/docker.sock:ro -v /etc/traefik/traefik.toml:/traefik.toml:ro traefik:v2.1.8

Finally, create a config file to proxy http connections to traefik by default. To do that, first create /etc/httpd/conf.d/default.conf and populate it with

<VirtualHost *:80>
    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

<VirtualHost *:443>
    ErrorLog /var/log/httpd/traefik-proxy-error_log
    CustomLog /var/log/httpd/traefik-proxy-access_log common
    ForensicLog /var/log/httpd/traefik-proxy-forensic_log
    ProxyPreserveHost On

    ProxyPass / http://localhost:50080/
    ProxyPassReverse / http://localhost:50080/

    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/www_server.crt
    SSLCertificateKeyFile /etc/pki/tls/private/www_server.key
    SSLCACertificateFile /etc/pki/tls/certs/cacert.crt
</VirtualHost>

Then edit /etc/httpd/conf/httpd.conf and add the following line BEFORE the IncludeOptional conf.d/*.conf line

IncludeOptional conf.d/default.conf

Finally, restart the httpd service. Your apache web server should now forward all incoming connections to the traefik container unless there is a virtualhost configured in /etc/httpd/conf.d directory for the target host.

Now we have a working proxy, but we have not told it to proxy anything. Let's fix that problem.

Container deployments¶

Running multiple containers in a deployment can be tricky. You would need to make multiple Dockerfiles, create separate images, manage their networks and a hundred potential things more. Luckily we can bundle these tasks together with a tool called docker compose . From the docker documentation:

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

For our lab, we will run a deployment with compose that consists of the traefik proxy, devops_lab and devops_lab_db containers. We will also add configuration to the file for traefik to know which address corresponds to which container. In addition, we will expose and password protect traefik dashboard.

First, stop and remove all containers we have created in this lab.

docker stop devops_lab devops_lab_db traefik
docker rm devops_lab devops_lab_db traefik

Now navigate to yout devops_lab directory and create a file called docker-compose.yml. Then populate the file with the following:

networks:
  devops_lab:
    name: devops_lab
    external: true

services:

  traefik:
    image: "registry.hpc.ut.ee/mirror/traefik:v2.1.8"
    container_name: "traefik"
    ports:
      - "50080:80"
      - "58080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/etc/traefik/traefik.toml:/traefik.toml:ro"
      - "/etc/http.passwd:/etc/http.passwd:ro"
    networks:
      - devops_lab
    labels:
      traefik.http.routers.dashboard.rule: Host(`traefik-dashboard.<vm_name>.sa.cs.ut.ee`)
      traefik.http.routers.dashboard.service: api@internal
      traefik.http.routers.dashboard.middlewares: auth
      traefik.http.middlewares.auth.basicauth.usersfile: /etc/http.passwd
    restart: always
  devops_lab_db:
    image: "registry.hpc.ut.ee/mirror/mariadb"
    container_name: "devops_lab_db"
    networks:
      - devops_lab
    environment:
      MYSQL_ROOT_PASSWORD: "<SOME PASSWORD FOR DB ROOT>"
    volumes:
      - "/home/centos/devops_lab/db_data:/var/lib/mysql:rw"
    healthcheck:
      test: ["CMD", "mariadb-admin", "ping", "-h", "127.0.0.1", "-p${MYSQL_ROOT_PASSWORD}"]
      interval: 1s
      timeout: 3s
      retries: 30
    restart: always
  devops_lab:
    image: "devops_lab"
    container_name: "devops_lab"
    networks:
      - devops_lab
    environment:
      DATABASE_ROOT_PASSWORD: "<SOME PASSWORD FOR DB ROOT>"
    labels:
      traefik.http.routers.devops_lab.rule: Host(`devops-lab.<vm_name>.sa.cs.ut.ee`)
      traefik.port: 5000
    depends_on:
      devops_lab_db:
        condition: service_healthy
    restart: always

As you can see you are taking the commands that you would assign to docker run and are assigning them in a yaml file. This deployment runs a traefik container with an accompanying devops_lab and devops_lab_db containers.The traefik labels:

• traefik.http.routers.dashboard.rule: Host(traefik-dashboard.<vm_name>.sa.cs.ut.ee) tells traefik proxy to forward requests coming to traefik-dashboard.<vm_name>.sa.cs.ut.eeto the dashboard application running inside traefik container
• traefik.http.routers.dashboard.service: api@internal Sets the target to forward the request to
• traefik.http.routers.dashboard.middlewares: auth Tells traefik to use authentication for this endpoint
• traefik.http.middlewares.auth.basicauth.usersfile: /etc/http.passwd Tells traefik to use /etc/http.passwd as the authentication source

For the devops_lab there are a bit less labels and these should be self explanatory by now.

Now create /etc/http.passwd and create a user and password to access traefik dashboard

sudo htpasswd -c /etc/http.passwd <A FREELY CHOSEN USERNAME>

Finally run docker compose up -d. In the same directory where your docker-compose.yml file is. You should see output about your services starting.

Now, before we can actually access the services from the browser, we also need to add traefik-dashboard.<vm_name>.sa.cs.ut.ee and devops-lab.<vm_name>.sa.cs.ut.ee to our DNS condiguration:

traefik-dashboard               IN              CNAME   <vm_name>.sa.cs.ut.ee.
devops-lab                      IN              CNAME   <vm_name>.sa.cs.ut.ee.

Pushing a container to a registry¶

Now we have built a container but that container is only available on the machine it was built on. Sometimes we need to make the containers we build publicly available. You can take the mariadb or traefik containers as examples - we did not build them, we pulled them from a public registry.

To prepare for the next lab, we need to make the devops_lab container publicly available. For this, we have prepared a container registry for you. Navigate to https://registry.sa.cs.ut.ee/ and create a new account there. As the username use your university username. You can use any password you like.

Then, after logging in, create a new Private project with your <vm_name> as the project name. Finally, click on the created project and using the hints in the PUSH COMMAND drop-down menu, push your devops_lab container to the registry (use latest as the TAG).