Intalling Kubernetes with cri-o inside flatcar Container Linux

47 min read

How to run containers without dockershim / containerd by installing cri-o with crun under flatcar Container Linux.

Container distributions

When I started to use docker on customers premises for all production workloads, I quickly felt the need for a small container oriented distribution to get the most of (low-end) bare-metal servers resources. CoreOS disrupted the container world by the minimalism of their distribution, its read-only file-system (no package manager) and its dual-partition rolling update strategy.

When it went discontinued after the purchase of Red Hat and then IBM, I turned logically to its fork Flatcar Container Linux. I then switched to the edge channel to use cri-o with the crun container engine because I found that pressure on resources was significantly lower with this duo.

But the edge channel also ended to being discontinued, and as I didn’t want to go back using docker, I started to install cri-o manually on top of the flatcar stable channel.

After 3 years using the combo in production without having encountered any major problem, I think it is time to write a quick post about the installation procedure to help those still afraid of leaving docker land.

Read-only file-system

As most of standard directories of flatcar are read-only, and the system partitions are intentionally minimal, I have a systemd raid10 mount in /opt :


[Unit] Description=Mount /opt Before=crio.service Before=systemd-journald.socket [Mount] What=/dev/md127 Where=/opt Type=ext4 [Install] RequiredBy=systemd-journald.socket RequiredBy=crio.service

from which the following symlinks are defined :

sudo mkdir -p /opt/var/lib/{containers,etcd,kubelet} sudo ln -s /opt/var/lib/containers /var/lib/containers sudo ln -s /opt/var/lib/etcd /var/lib/etcd sudo ln -s /opt/var/lib/kubelet /var/lib/kubelet


cri-o is advertised as a lightweight container runtime for Kubernetes, and aims to replace completely docker.

Install binaries

Get the latest static release for your architecture (amd64) with :

# can't rely on releases/latest to have the latest version so sort the releases by tag_name URL=$(curl -s | jq -r '. |= sort_by(.tag_name) | .[-1] | .assets[] | select(.name|test("amd64.*.tar.gz$")) | .browser_download_url') curl -sL $URL | tar xzvf -

The arquive includes the crun binary which is a lightweight and fastest replacement of containerd or runc written in C.

An install script exits in the extracted cri-o directory, but let’s install only what’s necessary manually :

cd cri-o # copy binaries sudo cp bin/* /opt/bin/ sudo mkdir -p /opt/cni/bin sudo cp cni-plugins/* /opt/cni/bin/ # copy configuration files sudo cp etc/crio-umount.conf /etc/ sudo mkdir -p /etc/crio/crio.conf.d sudo cp etc/crio.conf /etc/crio/ sudo cp etc/10-crun.conf /etc/crio/crio.conf.d/ sudo cp etc/crictl.yaml /etc/ # copy network configuration files sudo mkdir -p /etc/cni/net.d sudo cp contrib/11-crio-ipv4-bridge.conflist /etc/cni/net.d/ # copy containers configuration sudo mkdir /etc/containers sudo cp contrib/policy.json /etc/containers/ # install systemd unit after changing path sudo sh -c "sed 's:/usr/local/bin:/opt/bin:g' contrib/crio.service > /etc/systemd/system/crio.service"

Updating binaries

Updating is a little more complicated and needs a reboot with the crio unit disabled to be able to overwrite the binaries :

sudo systemctl disable kubelet crio sudo reboot

And after reboot :

# overwrite binaries sudo cp bin/* /opt/bin/ sudo cp cni-plugins/* /opt/cni/bin/ # enable and start units sudo systemctl enable kubelet crio sudo systemctl start kubelet crio

Additional files

You need to create manually this file to set up the containers’ storage space :


[storage] driver = "overlay" runroot = "/var/run/containers/storage" graphroot = "/var/lib/containers/storage" [storage.options] additionalimagestores = [] [storage.options.overlay] ignore_chown_errors = "true"

As well as this one to be able to pull containers images from some well known public registries :


[] registries = ['', '', '', ''] [registries.insecure] registries = [] [registries.block] registries = [] # this allows to use unqualified image names from the docker era unqualified-search-registries = [""]

To let cri-o find executables in /opt/bin (conmon, crun, …), create the following file which is loaded in the crio.service unit file :



Now it’s time to start and stop services :

# refresh systemd config sudo systemctl daemon-reload # stop/desactivate docker and containerd sudo systemctl stop docker containerd sudo systemctl mask docker containerd # enable/start crio sudo systemctl enable crio sudo systemctl start crio

You should now be able to use cri-o

sudo crio -v
crio version 1.25.1 Version: 1.25.1 GitCommit: afa0c576fcafc095e2827261e412fadabb016874 GitCommitDate: 2022-10-07T15:21:08Z GitTreeState: dirty BuildDate: 1980-01-01T00:00:00Z GoVersion: go1.18.1 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: static netgo osusergo exclude_graphdriver_btrfs exclude_graphdriver_devicemapper seccomp apparmor selinux LDFlags: -s -w -X"" -X -s -w -linkmode external -extldflags "-static -lm" SeccompEnabled: true AppArmorEnabled: false Dependencies:
sudo crio-status info
cgroup driver: systemd storage driver: overlay storage root: /opt/var/lib/containers/storage default GID mappings (format <container>:<host>:<size>): 0:0:4294967295 default UID mappings (format <container>:<host>:<size>): 0:0:4294967295


Flatcar Container Linux had a Kubernetes installation procedure (lokomotive) but it has been archived recently.

Hopefully the project it was based on (made by a core CoreOS developer) is still well and alive: Typhoon.

It allows bootstrapping bare-metal Kubernetes cluster nodes using :

  • an IPXE server,

  • a configuration server (matchbox), and

  • terraform to generate the certificates, the manifests and execute remotely ssh commands on the nodes to proceed with the installation.

I started my Kubernetes journey with Typhoon at a time when kubeadm didn’t exist and when self-hosted Kubernetes (i.e. control plane managed by itself) was still experimental with bootkube, but nowadays kubeadm is doing part of the same job with a much simpler setup (but less flexibility).

What follows is a manual installation of Kubernetes on top of an already installed flatcar system (you can use a pen drive for instance). The procedure for installing flatcar Container Linux using typhoon (pxe + matchbox) will be the subject of another blog post.

Install binaries

This will allow to install the latest stable version of kubernetes :

# get kubeadm, kubelet and kubectl RELEASE="$(curl -sSL" url="${RELEASE}/bin/linux/amd64" curl -L --remote-name-all $url/{kubeadm,kubelet,kubectl} chmod +x {kubeadm,kubelet,kubectl} sudo cp {kubeadm,kubelet,kubectl} /opt/bin/ # get kubelet service unit and change kubelet path PKG_RELEASE="$(curl -s | jq -r '.tag_name')" sudo sh -c "curl -sSL${PKG_RELEASE}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service | sed 's:/usr/bin:/opt/bin:g' > /etc/systemd/system/kubelet.service" sudo mkdir -p /etc/systemd/system/kubelet.service.d # get kubeadm dropin for kubelet sudo sh -c "curl -sSL${PKG_RELEASE}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf | sed 's:/usr/bin:/opt/bin:g' > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf"

kubeadm configuration

Now you need to provision a kubeadm configuration. You can see the defaults with :

kubeadm config print init-defaults
apiVersion: bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent name: node taints: null --- apiServer: timeoutForControlPlane: 4m0s apiVersion: certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} etcd: local: dataDir: /var/lib/etcd imageRepository: kind: ClusterConfiguration kubernetesVersion: 1.24.0 networking: dnsDomain: cluster.local serviceSubnet: scheduler: {}

To make it work with cri-o, you need to change criSocket to criSocket: /var/run/crio/crio.sock and change a few values to match your environment (see comments in YAML manifest) :


--- apiVersion: kind: InitConfiguration localAPIEndpoint: # set your node IP advertiseAddress: "your_node_ip" bindPort: 6443 nodeRegistration: # use crio instead of containerd criSocket: /var/run/crio/crio.sock # set hostname name: "your_node_name" # remove taints (test/dev) taints: [] --- apiVersion: kind: ClusterConfiguration kubernetesVersion: stable # set your control plane name (should resolve to your nodes ip) controlPlaneEndpoint: "your_control_plane_name:6443" # move flexvolume-dir to a rw dir (ro on flatcar) controllerManager: extraArgs: flex-volume-plugin-dir: /opt/libexec/kubernetes/kubelet-plugins/volume/exec extraVolumes: - name: flexvolume-dir hostPath: /opt/libexec/kubernetes/kubelet-plugins/volume/exec mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec --- # match crio default configuration apiVersion: kind: KubeletConfiguration cgroupDriver: systemd

Now you can initialize the first node with :

sudo kubeadm init --config=kubeadm-config.yaml --upload-certs

Add nodes to the cluster

You can later join node to the cluster with :

kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

To get the token and cert-hash, execute this on the first node :

# create a new join token valid 24h TOKEN=$(kubeadm token create) # create a join configuration you can use on nodes sudo kubeadm config print join-defaults | sed "s#criSocket:.*#criSocket: /var/run/crio/crio.sock#;s#token:.*#token: $TOKEN#"


podman is not strictly necessary with Kubernetes, but I found it useful to bootstrap container images manually in a node, as crictl has no load or save command.

There are some issues that prevent the release of official static binaries for podman, but you can find the static-podman project on GitHub which works for my use case.

url="" curl -sL $( curl -sL $( curl -sL $url | jq -r '. |= sort_by(.tag_name) | .[-1] | .assets[] | select(.name == "podman-linux-amd64.tar.gz") | .url' ) | jq -r '.browser_download_url' ) | tar xzvf - sudo cp podman-linux-amd64/usr/local/bin/podman /opt/bin/ sudo cp podman-linux-amd64/etc/cni/net.d/87-podman-bridge.conflist /etc/cni/net.d/

Now you should be able to start a pod outside Kubernetes :

sudo podman run -it -rm alpine -- sh


I use cilium as my CNI driver because it has been focused since the beginning on performance.

It offers unique features (like kube-proxy replacement or iptables less routing), has great filtering capabilities (DNS, L7), amazing observability features (hubble), and is backed by the 2 of the major cloud providers. It gets its superpowers from kernel eBPF.

I may write an entire blog post on cilium later, but for flatcar Linux installation all you need to do is provisioning a systemd mount unit for BPF file-system :


[Unit] Description=Cilium BPF mounts Documentation= DefaultDependencies=no [Mount] What=bpffs Where=/sys/fs/bpf Type=bpf Options=rw,nosuid,nodev,noexec,relatime,mode=700 [Install]


Related posts



Managing roles for PostgreSQL with Vault on Kubernetes

Vault has a database secret engine with a PostgreSQL driver that helps to create short-lived roles with random passwords for your database applications, but putting everything in production is not as simple as it seems.

40 min read



Building an alpine golden image

How to build an alpine image to base all your containers on.

27 min read



Building / consuming alpine Linux packages inside containers and images

How to build alpine Linux packages you can later install inside other alpine based containers or images

26 min read