Cassandra on OpenEBS in OpenShift

Cassandra:

Apache Cassandra is a free and open-source, distributed, wide column store, NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure

OpenEBS:

OpenEBS is the leading open-source project for container-attached and container-native storage on Kubernetes. OpenEBS adopts Container Attached Storage (CAS) approach, where each workload is provided with a dedicated storage controller. OpenEBS implements granular storage policies and isolation that enable users to optimize storage for each specific workload.

OpenShift:

Red Hat OpenShift is a multifaceted, open source container application platform from Red Hat Inc. for the development, deployment and management of applications.

Deploying Cassandra on OpenEBS:

Pre-requisites:

Storage pool:

OpenEBS provisions persistent volumes for containerized applications in its storage pool. The disks attached in a cluster can be obtained through the following command.

[root@master deployers]# oc get disks
NAME AGE
disk-0cfccca1c2b46241890953a725d6e5bd 80d
disk-0e570c2917f88f553460bfed72c33c5a 19d
disk-17f034148a30ef9ae2557cffda9ae869 20d
disk-2cb335326e30192286ce250a7e9562b8 80d
disk-42157bb4ec1762a13324453f57359964 81d
disk-42b4fb20cd36896dfc2a486b977363de 82d
disk-5ebe0072a9e937b28315006adf648dc2 82d
disk-5ffac6eccd389ed98a48e4ab3327e7e9 81d
disk-619def6e7b3147a7a57e9e31d71f2906 80d
disk-61efde57f8c657e83f23a3adbacfb08c 80d

You can use above disks to create storage pool of different types using following manifest. Create cstor-spc.yml using the below manifest and populate the required disks under diskList spec.

---
apiVersion: openebs.io/v1alpha1
kind: StoragePoolClaim
metadata:
name: cstor-pool
annotations:
cas.openebs.io/config: |
- name: PoolResourceRequests
value: |-
memory: 1Gi
cpu: 100m
- name: PoolResourceLimits
value: |-
memory: 4Gi
- name: AuxResourceLimits
value: |-
memory: 0.5Gi
cpu: 50m
spec:
name: cstor-pool
type: disk
maxPools: 5
poolSpec:
poolType: striped
cacheFile: /tmp/pool2.cache
overProvisioning: false
disks:
diskList:
- disk-0cfccca1c2b46241890953a725d6e5bd
- disk-2cb335326e30192286ce250a7e9562b8
- disk-619def6e7b3147a7a57e9e31d71f2906
- disk-61efde57f8c657e83f23a3adbacfb08c
- disk-99127001e6b6bc190b5169e42ebecad7

Apply the above manifest after updating disk details.

oc apply -f cstor-spc.yml

To use OpenEBS Storage by an application, a storage class should have been created in the K8s cluster updated with its provisioner and storage pool.

Create a storage class template storage-class.yml using the following configuration.

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cstor-pool
annotations:
openebs.io/cas-type: cstor
cas.openebs.io/config: |
- name: StoragePoolClaim
value: “cstor-pool”
- name: ReplicaCount
value: “1”
provisioner: openebs.io/provisioner-iscsi

Update the desired storage pool under StoragePoolClaim in above configuration and apply it to create storage class.

oc apply -f storage-class.yml

Cassandra deployment:

Lets deploy Cassandra as Kubernetes Statefulset. A statefulset in Kubernetes requires a headless service to provide network access to the pods it creates.

Create such headless service using the following manifest.

---
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
clusterIP: None
ports:
- port: 9042
selector:
app: cassandra

Apply the above manifest to create headless service.

oc apply -f cassandra-service.yml

Create cassandra-statefulset.yml as shown below which can create a Statefulset for Cassandra with three replicas.

---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: cassandra
labels:
app: cassandra
spec:
serviceName: cassandra
replicas: 3
selector:
matchLabels:
app: cassandra
template:
metadata:
labels:
app: cassandra
spec:
containers:
- name: cassandra
image: gcr.io/google-samples/cassandra:v11
imagePullPolicy: Always
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7001
name: tls-intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
resources:
limits:
cpu: “500m”
memory: 1Gi
requests:
cpu: “500m”
memory: 1Gi
securityContext:
capabilities:
add:
- IPC_LOCK
lifecycle:
preStop:
exec:
command: [“/bin/sh”, “-c”, “PID=$(pidof java) && kill $PID && while ps -p $PID > /dev/null; do sleep 1; done”]
env:
- name: MAX_HEAP_SIZE
value: 512M
- name: HEAP_NEWSIZE
value: 100M
- name: CASSANDRA_SEEDS
value: “cassandra-0.cassandra.cassandra-cstor.svc.cluster.local”
- name: CASSANDRA_CLUSTER_NAME
value: “K8Demo”
- name: CASSANDRA_DC
value: “DC1-K8Demo”
- name: CASSANDRA_RACK
value: “Rack1-K8Demo”
- name: CASSANDRA_AUTO_BOOTSTRAP
value: “false”
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
initialDelaySeconds: 15
timeoutSeconds: 5
volumeMounts:
- name: cassandra-claim
mountPath: /cassandra_data
volumeClaimTemplates:
- metadata:
name: cassandra-claim
annotations:
volume.beta.kubernetes.io/storage-class: cstor-sc
spec:
accessModes: [ “ReadWriteOnce” ]
resources:
requests:
storage: 5G

Make sure to update the storage class, namespace and volume claim name in the above configuration before applying it.

Apply the above configuration to create cassandra ring.

oc apply -f cassandra-statefulset.yml -n <namespace>

You can check the statefulset status through following command.

[root@master deployers]# oc get sts
NAME DESIRED CURRENT AGE
cassandra 3 3 3h

Ensure that the PVC is bound to a volume through following command.

oc get pvc -n <namespace>
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cassandra-claim-cassandra-0 Bound pvc-c0d14696–6fd8–11e9–8a2a-0050569331ce 5G RWO cstor-sc 3h
cassandra-claim-cassandra-1 Bound pvc-1c040419–6fd9–11e9–8a2a-0050569331ce 5G RWO cstor-sc 3h
cassandra-claim-cassandra-2 Bound pvc-969bc0a9–6fd9–11e9–8a2a-0050569331ce 5G RWO cstor-sc 3h

Ensure that the cassandra cluster is created using nodetool status command.

root@cassandra-2:/var/lib/cassandra# nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
— Address Load Tokens Owns (effective) Host ID Rack
UN 192.30.5.71 166.93 KiB 32 60.8% 93b884e0-fb26–4ef3–903c-782cdde6bf74 Rack1-K8Demo
UN 192.28.6.145 129.61 KiB 32 64.0% 1b7d60d2–7cbb-4445–8bcb-3389c0c7b0e4 Rack1-K8Demo
UN 192.29.6.160 107.04 KiB 32 75.3% 7d37c2e5–1929–4d01-a164-ef243a575ae3 Rack1-K8Demo

Write sample data:

kubectl exec -it cassandra-0 -- bash
root@cassandra-0:/# cqlsh

Connected to K8Demo at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.

cqlsh> CREATE KEYSPACE openebsdb WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 };
cqlsh> use openebsdb;
cqlsh:openebsdb> CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint);
cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone) VALUES(42,'giri', 'blr', '3278362832')