KubeCon + CloudNativeCon Europe 2018

“Cloud” is a word that’s heard a lot these days in companies both large and small. The cloud brings benefits such as fault-tolerant servers, dynamic scalability and lower infrastructural costs for companies, just to name a few. Reaping these benefits requires a solid understanding of the enabling technologies and tools that are out there. I’m Miro Eklund, Software Developer at Semantum Oy and I recently attended Kubernetes + Cloud Native Conference (link) in Copenhagen, Denmark, to do just that: To take a look at what technologies and tools other companies are using in production and see where the cloud movement is going next. In this blog post I’ll give you an overview of things discussed at KubeCon Europe 2018, with a short introduction to Docker Swarm and Kubernetes and the most discussed topic in KubeCon: Containers.

Introduction

Containers (wikipedia definition) provide a way to package and isolate applications, ensuring the environment required by the applications stays the same in development all the way to production, despite running on vastly different host machines. With containers, applications with dependencies to very old libraries can run alongside brand-new applications with incompatible dependencies, as each container is given a unique, isolated environment where the applications are executed in.

Containers are available for Linux and Windows. Docker Swarm and Kubernetes are container orchestrators that handle the networking, deployment, health-checking, load-balancing and much more of containers in a cluster of computers. A cluster of machines in this context contains worker and manager machines, where an individual machine is also known as a node.

KubeCon focused on Kubernetes and new technologies specifically for it, but still had plenty of general topics regarding container best practices, security and more. My interest was specifically in Windows containers and Docker Swarm, which is why I spent a fair share of time at the Azure and Docker booths during breaks from presentations.

Here’s what I learned at KubeCon Europe 2018.

Windows (process) containers in production

Elton Stoneman greeted me at the Docker booth and shared his experience on Windows containers. Apparently, companies are running process containers in production on their Windows Server 2016 instances, rather than the much more isolating Hyper-V containers. The main difference between these two types is that Hyper-V containers are light-weight Virtual Machines, rather than “proper” containers. I had previously thought that only Hyper-V offered proper isolation, so I asked what the security implications are. I was told process containers offer more than adequate isolation, even for programs that run custom user-defined code.

Container security

Speaking of isolation and security, Liz Rice mentioned in one of her Keynotes that 87% of Docker images on docker hub run as root, analyzed with a text-parsing tool that read the source Dockerfiles. This essentially gives the container privileges to run new containers on the host system, if it can access the proper host directory. I recommend watching her keynote “Running with Scissors” on the subject (link below). When building a docker image we should always make sure to change to a non-root user and change permissions of necessary files accordingly.

An easy way to stop the specific exploit demonstrated by Liz Rice is to use an image signing tool when creating and deploying new containers. One such tool is Notary, based on “The Update Framework” (TUF). This disallows unsigned or untrusted images from being run as containers.

Persistent Storage (databases)

Containers are ephemeral, meaning that all data stored on them disappears when they are removed. This ephemeral design allows hundreds of replicas of a container to be run in a cluster and easily cleaned up once they aren’t needed, such as for a short burst when traffic increases on a popular website. Databases, however, do not scale in the same way and have mostly remained rigid single-node applications. The “Vitess” project aims to make all SQL databases distributable across multiple nodes, making them dynamically scalable and fault-tolerant through replication. However, not all databases are design with scalability in mind, and so cloud infrastructures need to take this into account when designing their storage solutions.

Logging

Since containers are ephemeral, any logs written to them are discarded when the container is removed. In a regular production environment, containers are created, removed and re-created all the time, meaning that logfiles written to them are unobtainable. Containers are designed to write all their logs to STDOUT and STDERR, so that Docker Swarm and Kubernetes can read them from there. An appropriate additional microservice, such as fluentd will them collect the logs from the containers and allows them to be stored outside the life-time of individual containers.

Interesting projects to keep an eye on or try yourself

The ecosystem around containers has evolved rapidly and many new software tools have been introduced to make setting up, maintaining and monitoring clusters easy. Here are a list and a short introduction to many of the projects showcased at KubeCon that I found interesting:

  • NATS Messaging - A high throughput messaging server for microservices
  • Spiffe + Spire - A trust-establishing software that allows microservices during runtime to check the identity of other microservices and thus know that they’re the correct one and signed properly
  • HTCondor - A workload manager specifically made for compute-intensive batch-jobs
  • Microscanner - A free tool that scans docker images for package vulnerabilities
  • TSDB - A highly optimized time-series database for float-values
  • Fluentd - A software that’s between applications (containers) and databases that stores logs, allowing logs to be collected from various workloads and stored in various databases
  • Grafana - An application that allows monitoring of resources, containers and applications in a Kubernetes cluster
  • Envoy - Service proxy like NGINX, built for large clusters
  • Jaeger - Traces calls between micro services in a cluster, allowing proper debugging and monitoring of every request

Talks and presentations that caught my attention:

  1. Liz Rice “Running with Scissors”

    In this keynote, Liz Rice remonstrates the dangers of running as root and shows how a user can create their own malicious containers directly on the host.

  2. Ed King & Julz Friedman “Rootless Containers”

    In this talk we are given a look on the anatomy of containers, the things that are secure and the things that are still in the works.

  3. Sarah Wells “Migrating 150+ Microservices to Kubernetes”

    This keynote gives insight into why and how Financial Times changed their underlying microservice infrastructure from a custom-built solution to Kubernetes.

  4. Oliver Beattie “Anatomy of a Production Kubernetes Outage”

    In this keynote Oliver shows how a production-environment outage happened at Monzo Bank and how they recovered from it.

  5. Justin Cappos “Notary TUF Intro”

    This is a good introduction talk to signed images using Notary.

  6. James Strachan “Jenkins X”

    This is a good introduction talk to Jenkins X, i.e. Jenkins specifically built to leverage the power of Kubernetes.

  7. Daniel Eichten & Oliver Thylmann “Adidas Rollout of Kubernetes”

    Adidas’ story on migrating to Kubernetes.

  8. Mark McBride “Replacing NGINX with Envoy”

    Envoy is designed for containers, whereas NGINX is very powerful for single machines. This talk gives a short introduction on how Envoy can replace NGINX in large clusters.

  9. Ricardo Rocha & Clenimar Filemon “CERN Experiences with Multi Cloud Kubernetes”

    CERN uses huge clusters to run massive numbers of batch-jobs. They created a multi cloud Kubernetes cluster to provide them with the computational power they need.

  10. Ricardo Aravena “Different Container Runtimes”

    An overview of the different container runtimes out there.

  11. Laura Frank “Raft Consensus”

    Docker Swarm and Kubernetes managers manage their clusters with raft consensus. This presentation gives an easy-to-understand introduction to how decisions are made, and conflicts avoided in a raft.

  12. Zachary Arnold & Austin Adams “High Security as Scale”

    A short presentation on container security in a production environment at Ygrene.

All presentations and keynotes of KubeCon + CloudNativeCon Europe 2018 can be found here.