Deploy K3s cluster on K3OS with OpenEBS as persistent storage provider

5 min readSep 12, 2019

K3s is a lightweight ,easy to install Kubernetes distribution which is less resource intensive.

k3OS is a Linux distribution designed to remove as much as possible OS maintenance in a Kubernetes cluster. The goal of k3OS is to minimize the operating system overhead while running Kubernetes (k3s) cluster. k3OS Kernel is forked from Ubuntu, whereas the user-space binaries are derived from Alpine Linux.

Both K3OS and K3s are developed by rancher labs.

Installing K3OS

K3OS can be installed on any hypervisor of your choice. Here, we cover the procedure for deploying K3OS on VMware Vsphere environment.

  • Download K3OS ISO file from the latest release and create a virtual machine in VMware VSphere. Mount the ISO file into hypervisor and start a virtual machine.
  • Select Run k3OS LiveCD or Installation and press <ENTER>.
Install K3OS

The system will boot-up and gives you the login prompt.

K3OS login prompt
  • Login as rancher user without providing password.
  • Set the password for rancher user to enable connectivity from other machines by running sudo passwd rancher
Set password for rancher user
  • Now, we have to install K3OS into disk. This can be done by running the command sudo os-config
  • Choose the option 1.Install to disk . Answer the proceeding questions and provide rancher user’s password.
Install K3Os to disk
  • As part of above command execution, you can configure the host as either server or agent. Select 1.server to configure K3s master.
  • While configuring server, set cluster secret which would be used while joining nodes to the server. After successful installation and server reboot, check the cluster status.
Checking nodes

Configure agent

  • Follow the above steps till installing K3OS into disk in all the hosts that you want to be part of K3s cluster.
  • To configure kubernetes agent with K3OS, select the option 2. agent while running sudo os-config command. You need to provide URL of server and secret configured previously.
configure K3s agent

After performing this, k3s agent will be configured as follows and it will be added to the server.

Check the cluster configuration by checking the nodes,

k3os-32750 [~]$ kubectl get nodes
k3os-14539 Ready <none> 5m16s v1.14.1-k3s.4
k3os-32750 Ready <none> 49m v1.14.1-k3s.4

Now , you have setup the K3s kubernetes cluster with two nodes.

Deploying OpenEBS

OpenEBS is the most widely deployed and easy to use open source storage solution for Kubernetes.

Deploy OpenEBS by running the following command.

k3os-32750 [~]$ kubectl apply -f
namespace/openebs created
serviceaccount/openebs-maya-operator created created created
deployment.apps/maya-apiserver created
service/maya-apiserver-service created
deployment.apps/openebs-provisioner created
deployment.apps/openebs-snapshot-operator created
configmap/openebs-ndm-config created
daemonset.extensions/openebs-ndm created
deployment.apps/openebs-ndm-operator created
secret/admission-server-certs created
service/admission-server-svc created
deployment.apps/openebs-admission-server created created
deployment.apps/openebs-localpv-provisioner created

Check the OpenEBS components by running the following command.

k3os-32750 [~]$ kubectl get pods -n openebs
maya-apiserver-78c966c446-zpvhh 1/1 Running 2 101s
openebs-admission-server-66f46564f5-8sz8c 1/1 Running 0 101s
openebs-localpv-provisioner-698496cf9b-wkf95 1/1 Running 0 101s
openebs-ndm-9kt4n 0/1 ContainerCreating 0 101s
openebs-ndm-mxqcf 0/1 ContainerCreating 0 101s
openebs-ndm-operator-7fb4894546-d2whz 1/1 Running 1 101s
openebs-provisioner-7f9c99cf9-9jlgc 1/1 Running 0 101s
openebs-snapshot-operator-79f7d56c7d-tk24k 2/2 Running 0 101s

Note that openebs-ndmpods are not created successfully. This is due to the lack of udev support in K3OS.(

But you can use OpenEBS local PV to serve your application needs.

By default, OpenEBS deployment ports a list of storage classes which can be used as example for creating new storage classes. Check the storage classes created as part of OpenEBS deployment by running the following command.

k3os-32055 [~]$ kubectl get sc
openebs-device 57m
openebs-hostpath 57m
openebs-jiva-default 57m
openebs-snapshot-promoter 57m

The storage class openebs-hostpath can be used to create local PV on the path /var/openebs/localin your kubernetes node. You can either use the above storage class to create volume or create a new storage class specifying different path in the node by following the doc(

Deploy an application using OpenEBS local PV.

For illustration, let us deploy Cassandra statefulset application on K3s in such a way to consume OpenEBS local PV.

We can use openebs-hostpath storage class in cassandra-statefulset manifest and provision the volumes.

Download cassandra application manifest,


Update storage class to openebs-hostpath and replica count in the above cassandra-statefulset manifest file.

Now, deploy application by running the following command.

k3os-32750 [~]$ kubectl apply -f cassandra-service.yaml 
service/cassandra created
k3os-32750 [~]$ kubectl apply -f cassandra-statefulset.yaml
statefulset.apps/cassandra created

Check the application pods by running the following command.

k3os-32750 [~]$ kubectl get pods
cassandra-0 1/1 Running 0 6m16s
cassandra-1 1/1 Running 0 5m8s

Check the status of OpenEBS persistent volumes by running the following command.

k3os-32750 [~]$ kubectl get pv
pvc-36f1d3d3-cf9b-11e9-bbd8-000c29b74861 5G RWO Delete Bound default/cassandra-data-cassandra-1 openebs-hostpath 8m43s
pvc-dbbcc44f-cf99-11e9-bbd8-000c29b74861 5G RWO Delete Bound default/cassandra-data-cassandra-0 openebs-hostpath 18m

k3os-32750 [~]$ kubectl get pvc
cassandra-data-cassandra-0 Bound pvc-dbbcc44f-cf99-11e9-bbd8-000c29b74861 5G RWO openebs-hostpath 18m
cassandra-data-cassandra-1 Bound pvc-36f1d3d3-cf9b-11e9-bbd8-000c29b74861 5G RWO openebs-hostpath 8m54s

OpenEBS local PV will not be bound until the application pod is scheduled as its volumeBindingMode is set to WaitForFirstConsumer. Once the application pod is scheduled on a certain node, OpenEBS local persistent volume will be bound on that node.

Thus, we can install K3s and utilize OpenEBS as storage provisioner for stateful applications.