Deploy K3s cluster on K3OS with OpenEBS as persistent storage provider

K3s is a lightweight ,easy to install Kubernetes distribution which is less resource intensive.

k3OS is a Linux distribution designed to remove as much as possible OS maintenance in a Kubernetes cluster. The goal of k3OS is to minimize the operating system overhead while running Kubernetes (k3s) cluster. k3OS Kernel is forked from Ubuntu, whereas the user-space binaries are derived from Alpine Linux.

Both K3OS and K3s are developed by rancher labs.

Installing K3OS

K3OS can be installed on any hypervisor of your choice. Here, we cover the procedure for deploying K3OS on VMware Vsphere environment.

• Download K3OS ISO file from the latest release and create a virtual machine in VMware VSphere. Mount the ISO file into hypervisor and start a virtual machine.
• Select Run k3OS LiveCD or Installation and press <ENTER>.

Install K3OS

The system will boot-up and gives you the login prompt.

K3OS login prompt

• Login as rancher user without providing password.
• Set the password for rancher user to enable connectivity from other machines by running sudo passwd rancher

Set password for rancher user

• Now, we have to install K3OS into disk. This can be done by running the command sudo os-config
• Choose the option 1.Install to disk . Answer the proceeding questions and provide rancher user’s password.

Install K3Os to disk

• As part of above command execution, you can configure the host as either server or agent. Select 1.server to configure K3s master.
• While configuring server, set cluster secret which would be used while joining nodes to the server. After successful installation and server reboot, check the cluster status.

Checking nodes

Configure agent

• Follow the above steps till installing K3OS into disk in all the hosts that you want to be part of K3s cluster.
• To configure kubernetes agent with K3OS, select the option 2. agent while running sudo os-config command. You need to provide URL of server and secret configured previously.

configure K3s agent

After performing this, k3s agent will be configured as follows and it will be added to the server.

Check the cluster configuration by checking the nodes,

k3os-32750 [~]$ kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
k3os-14539   Ready    <none>   5m16s   v1.14.1-k3s.4
k3os-32750   Ready    <none>   49m     v1.14.1-k3s.4

Now , you have setup the K3s kubernetes cluster with two nodes.

Deploying OpenEBS

OpenEBS is the most widely deployed and easy to use open source storage solution for Kubernetes.

Deploy OpenEBS by running the following command.

k3os-32750 [~]$ kubectl apply -f https://openebs.github.io/charts/openebs-operator-1.1.0.yaml

namespace/openebs created
serviceaccount/openebs-maya-operator created
clusterrole.rbac.authorization.k8s.io/openebs-maya-operator created
clusterrolebinding.rbac.authorization.k8s.io/openebs-maya-operator created
deployment.apps/maya-apiserver created
service/maya-apiserver-service created
deployment.apps/openebs-provisioner created
deployment.apps/openebs-snapshot-operator created
configmap/openebs-ndm-config created
daemonset.extensions/openebs-ndm created
deployment.apps/openebs-ndm-operator created
secret/admission-server-certs created
service/admission-server-svc created
deployment.apps/openebs-admission-server created
validatingwebhookconfiguration.admissionregistration.k8s.io/validation-webhook-cfg created
deployment.apps/openebs-localpv-provisioner created

Check the OpenEBS components by running the following command.

k3os-32750 [~]$ kubectl get pods -n openebs
NAME                                           READY   STATUS              RESTARTS   AGE
maya-apiserver-78c966c446-zpvhh                1/1     Running             2          101s
openebs-admission-server-66f46564f5-8sz8c      1/1     Running             0          101s
openebs-localpv-provisioner-698496cf9b-wkf95   1/1     Running             0          101s
openebs-ndm-9kt4n                              0/1     ContainerCreating   0          101s
openebs-ndm-mxqcf                              0/1     ContainerCreating   0          101s
openebs-ndm-operator-7fb4894546-d2whz          1/1     Running             1          101s
openebs-provisioner-7f9c99cf9-9jlgc            1/1     Running             0          101s
openebs-snapshot-operator-79f7d56c7d-tk24k     2/2     Running             0          101s

Note that openebs-ndmpods are not created successfully. This is due to the lack of udev support in K3OS.(https://github.com/openebs/openebs/issues/2686)

But you can use OpenEBS local PV to serve your application needs.

By default, OpenEBS deployment ports a list of storage classes which can be used as example for creating new storage classes. Check the storage classes created as part of OpenEBS deployment by running the following command.

k3os-32055 [~]$ kubectl get sc
NAME                        PROVISIONER                                                AGE
openebs-device              openebs.io/local                                           57m
openebs-hostpath            openebs.io/local                                           57m
openebs-jiva-default        openebs.io/provisioner-iscsi                               57m
openebs-snapshot-promoter   volumesnapshot.external-storage.k8s.io/snapshot-promoter   57m

The storage class openebs-hostpath can be used to create local PV on the path /var/openebs/localin your kubernetes node. You can either use the above storage class to create volume or create a new storage class specifying different path in the node by following the doc(https://docs.openebs.io/1.0.0-RC1/docs/next/uglocalpv.html)

Deploy an application using OpenEBS local PV.

For illustration, let us deploy Cassandra statefulset application on K3s in such a way to consume OpenEBS local PV.

We can use openebs-hostpath storage class in cassandra-statefulset manifest and provision the volumes.

Download cassandra application manifest,

wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/cassandra/cassandra-statefulset.yaml
wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/cassandra/cassandra-service.yaml

Update storage class to openebs-hostpath and replica count in the above cassandra-statefulset manifest file.

Now, deploy application by running the following command.

k3os-32750 [~]$ kubectl apply -f cassandra-service.yaml 
service/cassandra created
k3os-32750 [~]$ kubectl apply -f cassandra-statefulset.yaml 
statefulset.apps/cassandra created

Check the application pods by running the following command.

k3os-32750 [~]$ kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          6m16s
cassandra-1   1/1     Running   0          5m8s

Check the status of OpenEBS persistent volumes by running the following command.

k3os-32750 [~]$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                STORAGECLASS       REASON   AGE
pvc-36f1d3d3-cf9b-11e9-bbd8-000c29b74861   5G         RWO            Delete           Bound    default/cassandra-data-cassandra-1   openebs-hostpath            8m43s
pvc-dbbcc44f-cf99-11e9-bbd8-000c29b74861   5G         RWO            Delete           Bound    default/cassandra-data-cassandra-0   openebs-hostpath            18m

k3os-32750 [~]$ kubectl get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
cassandra-data-cassandra-0   Bound    pvc-dbbcc44f-cf99-11e9-bbd8-000c29b74861   5G         RWO            openebs-hostpath   18m
cassandra-data-cassandra-1   Bound    pvc-36f1d3d3-cf9b-11e9-bbd8-000c29b74861   5G         RWO            openebs-hostpath   8m54s

OpenEBS local PV will not be bound until the application pod is scheduled as its volumeBindingMode is set to WaitForFirstConsumer. Once the application pod is scheduled on a certain node, OpenEBS local persistent volume will be bound on that node.

Thus, we can install K3s and utilize OpenEBS as storage provisioner for stateful applications.