Deploy K8s on bare metal with D2IQ-Konvoy and use OpenEBS for storage.

Konvoy:

Konvoy is one of the distributions of Kubernetes that pre-packages the necessary add-on services such as prometheus, EFK, grafana with the native Kubernetes cluster.

In this blog, let me walkthrough the steps for deploying Kubernetes cluster using D2IQ konvoy package on premise virtual machines.

Deployer host

It requires a deployer machine where the konvoy command line interface (CLI) will be executed. The deployer host should have the following packages installed.

1. Docker Desktop 18.09.2 or later
2. Kubectl 15.5 or later

In my case, installed a Centos-7 box which act as konvoy deployer. In this machine, installed Docker desktop and kubectl.

While installing Konvoy, it prompts to disable selinux on the deployer host. After disabling selinux, follow the below steps to install Konvoy.

1. Download and extract Konvoy package on the deployer host.

tar -xf konvoy_v0.x.x_linux.tar.bz2

2. After extracting, move the binaries under your user PATH:

mv ./konvoy_v0.x.0/* /usr/local/bin/

1. Check the konvoy version by running the following command.

$ konvoy — version
v0.6.0: Pulling from mesosphere/konvoy
f5d23c7fed46: Pull complete
…
Digest: sha256:4faf95ce405554a88769a00e89b07b0f2a1257f1dc84115e021f0aaa1ca79020
Status: Downloaded newer image for mesosphere/konvoy:v0.6.0
docker.io/mesosphere/konvoy:v0.6.0
{
“Version”: “v0.6.0”,
“BuildDate”: “Thu Jul 25 04:55:40 UTC 2019”
}

It pulls the necessary images when we execute konvoy command first time right after its installation.

Host Prerequisites.

Konvoy brings up the Kubernetes cluster on Centos. Therefore the virtual machines for K8s master and nodes should be brought up with Centos 7.

On all the hosts, swap and firewalld should be disabled.

Disabling swap.

Swap space can be disabled on those by running the following command.

swapoff -a

Swap space can be permanently disabled by commenting out the swap line in `/etc/fstab` file.

Disabling firewalld.

To disable firewalld, run the following command as root user.

systemctl disable firewalld
[root@dev1-d2iq-master@mayalabs ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

Make sure that the above operations be performed on all the hosts which form the Kubernetes cluster.

Ensure the connectivity between deployer host and the Kubernetes host machines are enabled thereby it can do ssh to each host machine.

Setting up the Cluster

Follow the below steps to create Kubernetes cluster in the deployer host.

• Create a directory where the skeleton for inventory file will be generated.

mkdir dev1-d2iq
cd dev1-d2iq

• Execute the below command to initialize Konvoy in the above directory.

[root@konvoydeployer dev1-d2iq]# konvoy init — provisioner=none
Created configuration file successfully!

As a result of above command execution, two files named `cluster.yml` and `inventory.yml` would be generated.

• Update the `inventory.yml` file with the master and worker node details as follows:

[root@konvoydeployer dev1-d2iq]# cat inventory.yaml
control-plane:
hosts:
10.34.1.110:
node:
hosts:
10.34.1.111:
node_pool: “worker”
10.34.1.112:
node_pool: “worker”
10.34.1.113:
node_pool: “worker”
10.34.1.114:
node_pool: “worker”
all:
vars:
ansible_port: 22
ansible_user: “root”
order: sorted

1. The default control plane load balancer for Konvoy is based on Keepalived. Update a virtual IP address in the same subnet of host machines and deployer host under `spec.kubernetes.controlplane.controlPlaneEndpointOverride`. Here, I use the IP address 10.34.1.109 which is reachable to all the hosts.
2. You can enable/disable the add-ons by editing addons list in `cluster.yaml` file.
3. The default load balancer service for addons is based on MetalLB.To use MetalLB for addon load balancing, Identify a range of virtual IP addresses from your networking infrastructure and update under metallb addon as follows:

- name: metallb
enabled: true
values: |-
configInline:
address-pools:
- name: default
protocol: layer2
addresses:
- 10.x.x.x–10.x.x.x

1. Enable passwordless ssh connectivity from deployer host to all the kubernetes hosts(Both Master and worker nodes).
2. After updating configuration in cluster.yaml file, run the Konvoy pre-flight checks to ensure that your on-premise environment has everything ready for installing Konvoy. Make sure that you run the below command from the directory where you have `cluster.yaml` and `inventory.yaml` files.

konvoy check preflight

• Finally deploy the cluster by running following command.

konvoy up

It will take upto 15 minutes to bootstrap the cluster along with selected add-on services.

After successful deployment, a file `admin.conf` will be generated in the same directory where we initiated installation. This is the kubeconfig file which can be used to access the cluster.

Once it is successfully bootstrapped, You can check the cluster status by running

giri@rack2:~/.kube$ kubectl cluster-info
Kubernetes master is running at https://10.34.x.x:6443
KubeDNS is running at https://10.34.x.x:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
kubernetes-dashboard is running at https://10.34.x.x:6443/api/v1/namespaces/kube-system/services/kubernetes-dashboard:http/proxy

To further debug and diagnose cluster problems, use `kubectl cluster-info dump`

Check the node status by running the following command.

giri@rack2:~/.kube$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
dev1-d21q-node2.mayalabs.io Ready <none> 27m v1.15.0
dev1-d2iq-master.mayalabs.io Ready master 28m v1.15.0
dev1-d2iq-node1.mayalabs.io Ready <none> 27m v1.15.0
dev1-d2iq-node3.mayalabs.io Ready <none> 27m v1.15.0
dev1-d2iq-node4.mayalabs.io Ready <none> 27m v1.15.0

Deploy OpenEBS

OpenEBS enables persistent volume provisioning for the workloads in Kubernetes cluster through its various storage engines such as jiva, cStor and localPV.

To deploy OpenEBS, run the following command.

kubectl apply -f https://openebs.github.io/charts/openebs-operator-1.6.0.yaml

Check if the OpenEBS components are installed successfully through below command.

kubectl get pods -n openebs

You can use any of the storage engines provided by OpenEBS to provision volumes dynamically for your stateful workloads.