Setting Up Elasticsearch and Kibana on Kubernetes: A Comprehensive Guide

In this extensive guide, we'll walk through the process of deploying both Elasticsearch and Kibana on Kubernetes. This setup will provide you with a powerful search and analytics stack that's scalable and easy to manage. We'll cover everything from creating the necessary Kubernetes resources to accessing your newly deployed services.

Prerequisites

Before we begin, ensure you have:

1. A Kubernetes cluster up and running

2. kubectl installed and configured to interact with your cluster

3. Basic knowledge of Kubernetes concepts

Part 1: Setting Up Elasticsearch

Step 1: Creating a StorageClass

First, we'll create a StorageClass that will be used to provision our Elasticsearch storage. This StorageClass is configured to use AWS EBS (Elastic Block Store) volumes.

Create a file named ebs-sc.yaml with the following content:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: existing-ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  csi.storage.k8s.io/fstype: ext4
  type: gp3
allowVolumeExpansion: true

Apply it using:

kubectl apply -f ebs-sc.yaml

Step 2: Creating Elasticsearch Resources

Now, let's create a file named es-service.yaml that will contain all the necessary resources for Elasticsearch. This includes a PersistentVolume, PersistentVolumeClaim, StatefulSet, and Service.

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: elasticsearch-data-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: existing-ebs-sc
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: vol-059ad22b5f6511b6d  # Replace with your EBS volume ID

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-data
  namespace: efk
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: existing-ebs-sc
  volumeName: elasticsearch-data-pv

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: efk
spec:
  serviceName: "elasticsearch"
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
        ports:
        - containerPort: 9200
          name: port1
        - containerPort: 9300
          name: port2
        env:
        - name: discovery.type
          value: single-node
        - name: xpack.security.enabled
          value: "false"
        volumeMounts:
        - name: es-data
          mountPath: /usr/share/elasticsearch/data
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: es-data
          mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: es-data
        persistentVolumeClaim:
          claimName: elasticsearch-data

---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: efk
spec:
  selector:
    app: elasticsearch
  ports:
  - port: 9200
    targetPort: 9200
    nodePort: 30200
    name: port1
  - port: 9300
    targetPort: 9300
    nodePort: 30300
    name: port2
  type: NodePort

Step 3: Deploying Elasticsearch

Now that we have all our configurations ready, let's deploy Elasticsearch:

1. First, create the efk namespace if it doesn't exist:

    kubectl create namespace efk
   

2. Apply the Elasticsearch configurations:

    kubectl apply -f es-service.yaml
   

3. Check the status of the Elasticsearch pod:

    kubectl get pods -n efk
   

   Wait until the pod is in the Running state.

Part 2: Setting Up Kibana

Now that we have Elasticsearch up and running, let's deploy Kibana to visualize and interact with our data.

Step 1: Creating Kibana Resources

Create a file named kibana.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: efk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:8.13.0
        ports:
        - containerPort: 5601
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: efk
spec:
  type: LoadBalancer
  selector:
    app: kibana
  ports:
  - protocol: TCP
    port: 5601
    targetPort: 5601

This configuration creates a Kibana Deployment and a LoadBalancer Service to expose Kibana externally.

Step 2: Deploying Kibana

Now, let's deploy Kibana:

1. Apply the Kibana configurations:

    kubectl apply -f kibana.yaml
   

2. Check the status of the Kibana pod:

    kubectl get pods -n efk
   

   Wait until the Kibana pod is in the Running state.

3. Get the external IP address of the Kibana service:

    kubectl get services -n efk
   

   Look for the EXTERNAL-IP of the kibana service. It may take a few minutes for the LoadBalancer to provision an external IP.

Accessing Your Elasticsearch and Kibana Setup

1. Elasticsearch:

   • Within the cluster: http://elasticsearch.efk.svc.cluster.local:9200

   • Externally: http://<Node-IP>:30200

2. Kibana:

   • Externally: http://<EXTERNAL-IP>:5601

Replace <Node-IP> with the IP of any of your Kubernetes nodes, and <EXTERNAL-IP> with the external IP address of the Kibana LoadBalancer service.

Conclusion

Congratulations! You've successfully set up both Elasticsearch and Kibana on Kubernetes. This powerful combination provides you with a robust search engine and a beautiful interface for data visualization and exploration.

Here's a quick recap of what we've accomplished:

1. Created a StorageClass for Elasticsearch data persistence

2. Deployed Elasticsearch as a StatefulSet with persistent storage

3. Exposed Elasticsearch within the cluster and externally

4. Deployed Kibana as a Deployment

5. Exposed Kibana externally using a LoadBalancer

Some next steps you might consider:

1. Configuring Elasticsearch for high availability with multiple nodes

2. Setting up index patterns in Kibana

3. Implementing security measures like enabling X-Pack security

4. Setting up log ingestion using Filebeat or Logstash

Remember to adjust these configurations based on your specific requirements, especially for production environments. Always ensure you're following best practices for security and performance.

Happy searching and visualizing!