Running Elasticsearch, Kibana & Filebeat In Kubernetes

Here I show you how you deploy Elasticsearch, Kibana & Filebeat in Kubernetes.

elastic

Note you have to upgrade to any cloud provider. If you wanted to run Elasticsearch in Minikube then It will give an ErrImagePull error.

Now it's time to install and run Elasticsearch, Kibana & Filebeat.

ECK

—

Now install ECK (Elastic Cloud on Kubernetes) resources

kubectl create -f https://download.elastic.co/downloads/eck/2.5.0/crds.yaml

Then

kubectl apply -f https://download.elastic.co/downloads/eck/2.5.0/operator.yaml

After installing those files then, Kubernetes will create a new namespace.

elastic-system

After some time it will be running. we can see the elastic-system status

kubectl get all -n elastic-system

Output

NAME                     READY   STATUS    RESTARTS   AGE
pod/elastic-operator-0   1/1     Running   0          40s
NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/elastic-webhook-server   ClusterIP   10.245.253.11   <none>        443/TCP   41s
NAME                                READY   AGE
statefulset.apps/elastic-operator   1/1     41s

Now the elastic ECK running well.

Elasticsearch

—

Now need to install Elasticsearch.

At the terminal type or you can create a elastic-search.yaml file

cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.5.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
EOF

If you create a file then

kubectl apply -f elasticsearch.yaml

Elasticsearch runs in Kubernetes.

NAME                                 READY   STATUS    RESTARTS   AGE
pod/quickstart-es-default-0          1/1     Running   0          68m

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                    ClusterIP   10.245.0.1       <none>        443/TCP    75m
service/quickstart-es-default         ClusterIP   None             <none>        9200/TCP   68m
service/quickstart-es-http            ClusterIP   10.245.182.187   <none>        9200/TCP   68m
service/quickstart-es-internal-http   ClusterIP   10.245.45.122    <none>        9200/TCP   68m
service/quickstart-es-transport       ClusterIP   None             <none>        9300/TCP   68m
NAME                                     READY   AGE
statefulset.apps/quickstart-es-default   1/1     68m

After that, you need to check the Elasticsearch at the browser

kubectl port-forward service/quickstart-es-http 9200

Here is the important things arise if you visit the url with

http://localhost:9200

Then it can't access the site. It will show this

error image

If you access with

https://localhost:9200

But you have to access the key to access the site. when you go into 1st time then it will ask for access.

Here you have to find the access from

secrets/quickstart-es-elastic-user

You can find the access from Lens

For that access of username and password

Username: elastic
Password: <From At the Lens>

After the username and password. you show this.

Elasticsearch running successfully.

Kibana

—

Now for Kibana.

At the terminal type

cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 8.5.0
  count: 1
  elasticsearchRef:
    name: quickstart
EOF

If you wanted to create a kibana.yaml then

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 8.5.0
  count: 1
  elasticsearchRef:
    name: quickstart

Then type

kubectl apply -f kibana.yaml

Output

NAME                                 READY   STATUS    RESTARTS   AGE
pod/quickstart-es-default-0          1/1     Running   0          68m
pod/quickstart-kb-697cdc8cb7-vvzqg   1/1     Running   0          19m
NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                    ClusterIP   10.245.0.1       <none>        443/TCP    75m
service/quickstart-es-default         ClusterIP   None             <none>        9200/TCP   68m
service/quickstart-es-http            ClusterIP   10.245.182.187   <none>        9200/TCP   68m
service/quickstart-es-internal-http   ClusterIP   10.245.45.122    <none>        9200/TCP   68m
service/quickstart-es-transport       ClusterIP   None             <none>        9300/TCP   68m
service/quickstart-kb-http            ClusterIP   10.245.162.67    <none>        5601/TCP   19m
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/quickstart-kb   1/1     1            1           19m
NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/quickstart-kb-697cdc8cb7   1         1         1       19m
NAME                                     READY   AGE
statefulset.apps/quickstart-es-default   1/1     68m

Now need to check if the Kibana is running or not.

At terminal type

kubectl port-forward service/quickstart-kb-http 5601

Here is also the same way to access the site.

https://localhost:5601/

In the browser, Kibana is accessible for password

Kibana login page

Use the same username and password

Username: elastic
Password: <From At the Lens>

Kibana running Done.

Filebeat

—

Now install Filebeat for showing Kubernetes logs.

It's actually a different process. 1st download the Filebeat Yaml file

curl -L -O https://raw.githubusercontent.com/elastic/beats/8.5/deploy/kubernetes/filebeat-kubernetes.yaml

Now need some changes for running Filebeat.

Enter the filebeat-kubernetes.yaml file

Here is the list of change

1. Add the certificate
2. Mapping the es (elastic search) hostname

Adding the certificate and Mapping the hostname

Change the namespace at every place

namespace: default

kind: ConfigMap

output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl.certificate_authorities:
- /etc/certificate/ca.crt

kind: DaemonSet

env:
- name: ELASTICSEARCH_HOST
value: https://quickstart-es-http 
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: ckg45J3h44XftuRe91x394Il
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName

Note: Here is the quickstart-es-http (service/quickstart-es-http)

volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: certs
mountPath: /etc/certificate/ca.crt
readOnly: true
subPath: ca.crt
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true

Config done.

Here is the reference of filebeat-kubernetes.yaml

Now need to config the Firebeat for getting the logs

Now go to Management. Then click Data View.

Now Create data view

Then click the button Save data to Kibana.

Then go to the Analytics slider then click Discover

Now here Analytics shows all data logs

Happy Learning!