How to run containers without dockershim / containerd by installing cri-o with crun under flatcar Container Linux.

Container distributions

When I started to use docker on customers premises for all production workloads, I quickly felt the need for a small container oriented distribution to get the most of (low-end) bare-metal servers resources. CoreOS disrupted the container world by the minimalism of their distribution, its read-only file-system (no package manager) and its dual-partition rolling update strategy.

When it went discontinued after the purchase of Red Hat and then IBM, I turned logically to its fork Flatcar Container Linux. I then switched to the edge channel to use cri-o with the crun container engine because I found that pressure on resources was significantly lower with this duo.

But the edge channel also ended to being discontinued, and as I didn’t want to go back using docker, I started to install cri-o manually on top of the flatcar stable channel.

After 3 years using the combo in production without having encountered any major problem, I think it is time to write a quick post about the installation procedure to help those still afraid of leaving docker land.

Read-only file-system

As most of standard directories of flatcar are read-only, and the system partitions are intentionally minimal, I have a systemd raid10 mount in /opt :

/etc/systemd/system/opt.mount

[Unit]
Description=Mount /opt
Before=crio.service
Before=systemd-journald.socket

[Mount]
What=/dev/md127
Where=/opt
Type=ext4

[Install]
RequiredBy=systemd-journald.socket
RequiredBy=crio.service

from which the following symlinks are defined :

sudo mkdir -p /opt/var/lib/{containers,etcd,kubelet}
sudo ln -s /opt/var/lib/containers /var/lib/containers
sudo ln -s /opt/var/lib/etcd /var/lib/etcd
sudo ln -s /opt/var/lib/kubelet /var/lib/kubelet

Cri-o

cri-o is advertised as a lightweight container runtime for Kubernetes, and aims to replace completely docker.

Install binaries

Get the latest static release for your architecture (amd64) with :

# can't rely on releases/latest to have the latest version so sort the releases by tag_name 
URL=$(curl -s https://api.github.com/repos/cri-o/cri-o/releases | jq -r '. |= sort_by(.tag_name) | .[-1] | .assets[] | select(.name|test("amd64.*.tar.gz$")) | .browser_download_url')
curl -sL $URL | tar xzvf -

The arquive includes the crun binary which is a lightweight and fastest replacement of containerd or runc written in C.

An install script exits in the extracted cri-o directory, but let’s install only what’s necessary manually :

cd cri-o
# copy binaries
sudo cp bin/* /opt/bin/
sudo mkdir -p /opt/cni/bin
sudo cp cni-plugins/* /opt/cni/bin/
# copy configuration files
sudo cp etc/crio-umount.conf /etc/
sudo mkdir -p /etc/crio/crio.conf.d
sudo cp etc/crio.conf /etc/crio/
sudo cp etc/10-crun.conf /etc/crio/crio.conf.d/
sudo cp etc/crictl.yaml /etc/
# copy network configuration files
sudo mkdir -p /etc/cni/net.d
sudo cp contrib/11-crio-ipv4-bridge.conflist /etc/cni/net.d/
# copy containers configuration
sudo mkdir /etc/containers
sudo cp contrib/policy.json /etc/containers/
# install systemd unit after changing path
sudo sh -c "sed 's:/usr/local/bin:/opt/bin:g' contrib/crio.service  > /etc/systemd/system/crio.service"

Updating binaries

Updating is a little more complicated and needs a reboot with the crio unit disabled to be able to overwrite the binaries :

sudo systemctl disable kubelet crio
sudo reboot

And after reboot :

# overwrite binaries
sudo cp bin/* /opt/bin/
sudo cp cni-plugins/* /opt/cni/bin/
# enable and start units
sudo systemctl enable kubelet crio 
sudo systemctl start kubelet crio

Additional files

You need to create manually this file to set up the containers’ storage space :

/etc/containers/storage.conf

[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = []
[storage.options.overlay]
ignore_chown_errors = "true"

As well as this one to be able to pull containers images from some well known public registries :

/etc/containers/registries.conf

[registries.search]
registries = ['docker.io', 'registry.fedoraproject.org', 'registry.access.redhat.com', 'quay.io']
[registries.insecure]
registries = []
[registries.block]
registries = []
# this allows to use unqualified image names from the docker era
unqualified-search-registries = ["docker.io"]

To let cri-o find executables in /opt/bin (conmon, crun, …), create the following file which is loaded in the crio.service unit file :

/etc/sysconfig/crio

PATH=/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

Now it’s time to start and stop services :

# refresh systemd config
sudo systemctl daemon-reload
# stop/desactivate docker and containerd
sudo systemctl stop docker containerd
sudo systemctl mask docker containerd
# enable/start crio
sudo systemctl enable crio
sudo systemctl start crio

You should now be able to use cri-o

sudo crio -v

crio version 1.25.1
Version:        1.25.1
GitCommit:      afa0c576fcafc095e2827261e412fadabb016874
GitCommitDate:  2022-10-07T15:21:08Z
GitTreeState:   dirty
BuildDate:      1980-01-01T00:00:00Z
GoVersion:      go1.18.1
Compiler:       gc
Platform:       linux/amd64
Linkmode:       static
BuildTags:      
  static
  netgo
  osusergo
  exclude_graphdriver_btrfs
  exclude_graphdriver_devicemapper
  seccomp
  apparmor
  selinux
LDFlags:          -s -w -X github.com/cri-o/cri-o/internal/pkg/criocli.DefaultsPath="" -X github.com/cri-o/cri-o/internal/version.buildDate=1980-01-01T00:00:00Z -s -w -linkmode external -extldflags "-static -lm"
SeccompEnabled:   true
AppArmorEnabled:  false
Dependencies:

sudo crio-status info

cgroup driver: systemd
storage driver: overlay
storage root: /opt/var/lib/containers/storage
default GID mappings (format <container>:<host>:<size>):
  0:0:4294967295
default UID mappings (format <container>:<host>:<size>):
  0:0:4294967295

Kubernetes

Flatcar Container Linux had a Kubernetes installation procedure (lokomotive) but it has been archived recently.

Typhoon

Hopefully the project it was based on (made by a core CoreOS developer) is still well and alive: Typhoon.

It allows bootstrapping bare-metal Kubernetes cluster nodes using :

an IPXE server,
a configuration server (matchbox), and
terraform to generate the certificates, the manifests and execute remotely ssh commands on the nodes to proceed with the installation.

I started my Kubernetes journey with Typhoon at a time when kubeadm didn’t exist and when self-hosted Kubernetes (i.e. control plane managed by itself) was still experimental with bootkube, but nowadays kubeadm is doing part of the same job with a much simpler setup (but less flexibility).

What follows is a manual installation of Kubernetes on top of an already installed flatcar system (you can use a pen drive for instance). The procedure for installing flatcar Container Linux using typhoon (pxe + matchbox) will be the subject of another blog post.

Install binaries

This will allow to install the latest stable version of kubernetes :

# get kubeadm, kubelet and kubectl
RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
url="https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64"
curl -L --remote-name-all $url/{kubeadm,kubelet,kubectl}
chmod +x {kubeadm,kubelet,kubectl}
sudo cp {kubeadm,kubelet,kubectl} /opt/bin/

# get kubelet service unit and change kubelet path
PKG_RELEASE="$(curl -s https://api.github.com/repos/kubernetes/release/releases/latest | jq -r '.tag_name')"
sudo sh -c "curl -sSL https://raw.githubusercontent.com/kubernetes/release/${PKG_RELEASE}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service | sed 's:/usr/bin:/opt/bin:g' > /etc/systemd/system/kubelet.service"
sudo mkdir -p /etc/systemd/system/kubelet.service.d
# get kubeadm dropin for kubelet
sudo sh -c "curl -sSL https://raw.githubusercontent.com/kubernetes/release/${PKG_RELEASE}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf | sed 's:/usr/bin:/opt/bin:g' > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf"

kubeadm configuration

Now you need to provision a kubeadm configuration. You can see the defaults with :

kubeadm config print init-defaults

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
  - groups:
      - system:bootstrappers:kubeadm:default-node-token
    token: abcdef.0123456789abcdef
    ttl: 24h0m0s
    usages:
      - signing
      - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: 1.24.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

To make it work with cri-o, you need to change criSocket to criSocket: /var/run/crio/crio.sock and change a few values to match your environment (see comments in YAML manifest) :

kubeadm.yaml

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  # set your node IP
  advertiseAddress: "your_node_ip"
  bindPort: 6443
nodeRegistration:
  # use crio instead of containerd
  criSocket: /var/run/crio/crio.sock
  # set hostname
  name: "your_node_name"
  # remove taints (test/dev)
  taints: []
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
# set your control plane name (should resolve to your nodes ip)
controlPlaneEndpoint: "your_control_plane_name:6443"
# move flexvolume-dir to a rw dir (ro on flatcar)
controllerManager:
  extraArgs:
    flex-volume-plugin-dir: /opt/libexec/kubernetes/kubelet-plugins/volume/exec
  extraVolumes:
    - name: flexvolume-dir
      hostPath: /opt/libexec/kubernetes/kubelet-plugins/volume/exec
      mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
---
# match crio default configuration
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Now you can initialize the first node with :

sudo kubeadm init --config=kubeadm-config.yaml --upload-certs

Add nodes to the cluster

You can later join node to the cluster with :

kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

To get the token and cert-hash, execute this on the first node :

# create a new join token valid 24h
TOKEN=$(kubeadm token create)
# create a join configuration you can use on nodes
sudo kubeadm config print join-defaults |
sed "s#criSocket:.*#criSocket: /var/run/crio/crio.sock#;s#token:.*#token: $TOKEN#"

podman

podman is not strictly necessary with Kubernetes, but I found it useful to bootstrap container images manually in a node, as crictl has no load or save command.

There are some issues that prevent the release of official static binaries for podman, but you can find the static-podman project on GitHub which works for my use case.

url="https://api.github.com/repos/mgoltzsche/podman-static/releases"
curl -sL $(
  curl -sL $(
    curl -sL $url |
    jq -r '. |= sort_by(.tag_name) | .[-1] | .assets[] | select(.name == "podman-linux-amd64.tar.gz") | .url'
  ) |
  jq -r '.browser_download_url'
) |
tar xzvf -
sudo cp podman-linux-amd64/usr/local/bin/podman /opt/bin/
sudo cp podman-linux-amd64/etc/cni/net.d/87-podman-bridge.conflist /etc/cni/net.d/

Now you should be able to start a pod outside Kubernetes :

sudo podman run -it -rm alpine -- sh

cilium

I use cilium as my CNI driver because it has been focused since the beginning on performance.

It offers unique features (like kube-proxy replacement or iptables less routing), has great filtering capabilities (DNS, L7), amazing observability features (hubble), and is backed by the 2 of the major cloud providers. It gets its superpowers from kernel eBPF.

I may write an entire blog post on cilium later, but for flatcar Linux installation all you need to do is provisioning a systemd mount unit for BPF file-system :

/etc/systemd/system/sys-fs-bpf.mount

[Unit]
Description=Cilium BPF mounts
Documentation=https://docs.cilium.io/
DefaultDependencies=no
Before=local-fs.target umount.target
After=swap.target

[Mount]
What=bpffs
Where=/sys/fs/bpf
Type=bpf
Options=rw,nosuid,nodev,noexec,relatime,mode=700

[Install]
WantedBy=multi-user.target

Éric BURGHARD

Intalling Kubernetes with cri-o inside flatcar Container Linux

Container distributions

Read-only file-system

Cri-o

Install binaries

Updating binaries

Additional files

Kubernetes

Install binaries

kubeadm configuration

Add nodes to the cluster

podman

cilium

Related posts

Comments

Intalling Kubernetes with cri-o inside flatcar Container Linux

Container distributions#

Read-only file-system#

Cri-o#

Install binaries#

Updating binaries#

Additional files#

Kubernetes#

Install binaries#

kubeadm configuration#

Add nodes to the cluster#

podman#

cilium#

Related posts

Comments

Container distributions

Read-only file-system

Cri-o

Install binaries

Updating binaries

Additional files

Kubernetes

Install binaries

kubeadm configuration

Add nodes to the cluster

podman

cilium