Elastic Search and Kibana on Kubernetes on Portworx

These below instructions will provide you with a step by step guide in deploying the Elasticstack (Elastic Search, Logstash, Kibana) with Portworx on Kubernetes.

Elasticsearch creates a cluster based on the cluster name property specified in the configuration. Each node within the cluster can forward client requests to the appropriate node and also knows about every other node in the cluster.

An Elasticsearch cluster node can have one or more purposes:

Master-eligible node A node that has node.master set to true (default), which makes it eligible to be elected as the master node, which controls the cluster.
Data node A node that has node.data set to true (default). Data nodes hold data and perform data related operations such as CRUD, search, and aggregations.
Coordinating node A node which only routes requests, handles the search reduce phase, and distributes bulk indexing. Consumers of the elastic search cluster interact with the coordinator nodes.

Prerequisites

A running Kubernetes cluster with v 1.6+
Deploy Portworx on your kubernetes cluster. Portworx runs on each node of your Kubernetes cluster as a DaemonSet.

Install

Portworx StorageClass for Volume Provisioning

Portworx provides volume(s) to the elastic search data and master nodes. Create portworx-sc.yaml with Portworx as the provisioner and apply the configuration. These storage classes create the Portworx volumes with 2 replicas when referenced via a PersistentVolumeClaim.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: px-es-data-sc
provisioner: kubernetes.io/portworx-volume
parameters:
   repl: "2"
   group: "elastic_data_vg"
   fg: "false"
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: px-es-master-sc
provisioner: kubernetes.io/portworx-volume
parameters:
   repl: "2"
   group: "elastic_master_vg"
   fg: "false"
---

Apply the spec by running:

kubectl apply -f portworx-sc.yaml

Install Elasticsearch cluster

In this section we will create an ES cluster with the following:

3 master nodes using a Kubernetes StatefulSet backed by Portworx volumes
3 data nodes using a Kubernetes StatefulSet backed by Portworx volumes
2 coordinator nodes using a Kubernetes Deployment

All pods will use the Stork scheduler to enable them to be placed closer to where their data is located.

Create es-master-svc.yaml with the following content

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-discovery
  labels:
    component: elasticsearch
    role: master
spec:
  selector:
    component: elasticsearch
    role: master
  ports:
  - name: transport
    port: 9300
    protocol: TCP

Create es-master-sts.yaml with the following content:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-master
  labels:
    component: elasticsearch
    role: master
spec:
  serviceName: elasticsearch-master
  selector:
    matchLabels:
     component: elasticsearch
  replicas: 3
  template:
    metadata:
      labels:
        component: elasticsearch
        role: master
    spec:
      # Use the stork scheduler to enable more efficient placement of the pods
      schedulerName: stork
      initContainers:
      - name: init-sysctl
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch-master-pod
        securityContext:
          privileged: true
          capabilities:
            add:
              - IPC_LOCK
              - SYS_RESOURCE
        image: quay.io/pires/docker-elasticsearch-kubernetes:6.3.2
        imagePullPolicy: Always
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: "CLUSTER_NAME"
          value: "escluster"
        - name: "NUMBER_OF_MASTERS"
          value: "2"
        - name: NODE_MASTER
          value: "true"
        - name: NODE_INGEST
          value: "false"
        - name: NODE_DATA
          value: "false"
        - name: HTTP_ENABLE
          value: "false"
        - name: "ES_JAVA_OPTS"
          value: "-Xms256m -Xmx256m"
        ports:
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: px-storage
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: px-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: px-es-master-sc
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 80Gi

Apply the specification for the Elastic search Master nodes and the service for the same.

kubectl apply -f es-master-svc.yaml

service "elasticsearch-discovery" created

kubectl apply -f es-master-sts.yaml

statefulset "es-master" created

kubectl get pods

NAME                         READY     STATUS            RESTARTS   AGE
elasticsearch-master-0       0/1       PodInitializing   0          22s
elasticsearch-master-1       0/1       PodInitializing   0          22s
elasticsearch-master-2       0/1       PodInitializing   0          22s

Verify that the master nodes have create and joined the cluster:

kubectl logs po/elasticsearch-master-0

Create es-coordinator-svc.yaml with the following content:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    component: elasticsearch
    role: coordinator
spec:
  selector:
    component: elasticsearch
    role: coordinator
  ports:
  - name: http
    port: 9200
    protocol: TCP

Create es-coordinator-deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch-coordinator
  labels:
    component: elasticsearch
    role: coordinator
spec:
  selector:
    matchLabels:
      component: elasticsearch
  replicas: 2
  template:
    metadata:
      labels:
        component: elasticsearch
        role: coordinator
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch-coordinator
        securityContext:
          privileged: false
          capabilities:
            add:
              - IPC_LOCK
              - SYS_RESOURCE
        image: quay.io/pires/docker-elasticsearch-kubernetes:6.3.2
        imagePullPolicy: Always
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: "CLUSTER_NAME"
          value: "escluster"
        - name: NODE_MASTER
          value: "false"
        - name: NODE_DATA
          value: "false"
        - name: HTTP_ENABLE
          value: "true"
        - name: "ES_JAVA_OPTS"
          value: "-Xms256m -Xmx256m"
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: storage
          mountPath: /data
      volumes:
          - emptyDir:
              medium: ""
            name: "storage"

Apply the specification for the coordinator Deployment and its service:

kubectl apply -f es-coordinator-deployment.yaml

deployment "elasticsearch-coordinator" created

kubectl apply -f es-coordinator-svc.yaml

service "elasticsearch" created

kubectl get pods -w

NAME                                            READY     STATUS    RESTARTS   AGE
po/elasticsearch-coordinator-2155074821-nxdkt   1/1       Running   0          1m
po/elasticsearch-coordinator-2155074821-v0w31   1/1       Running   0          1m
po/elasticsearch-master-0                       1/1       Running   0          4m
po/elasticsearch-master-1                       1/1       Running   0          4m
po/elasticsearch-master-2                       1/1       Running   0          4m

Verify that the coordinator nodes have joined the cluster:

kubectl logs po/elasticsearch-coordinator-2155074821-nxdkt

Create es-data-svc.yaml with the following content:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-data
  labels:
    component: elasticsearch
    role: data
spec:
  clusterIP: None
  selector:
    component: elasticsearch
    role: data
  ports:
  - name: transport
    port: 9300
    protocol: TCP

Create es-data-sts.yaml with the following content:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-data
  labels:
    component: elasticsearch
    role: data
spec:
  selector:
    matchLabels:
      component: elasticsearch
  serviceName: elasticsearch-data
  replicas: 3
  template:
    metadata:
      labels:
        component: elasticsearch
        role: data
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch-data-pod
        securityContext:
          privileged: true
          capabilities:
            add:
              - IPC_LOCK
        image: quay.io/pires/docker-elasticsearch-kubernetes:6.3.2
        imagePullPolicy: Always
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: "CLUSTER_NAME"
          value: "escluster"
        - name: NODE_MASTER
          value: "false"
        - name: NODE_INGEST
          value: "false"
        - name: HTTP_ENABLE
          value: "false"
        - name: "ES_JAVA_OPTS"
          value: "-Xms256m -Xmx256m"
        ports:
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: px-storage
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: px-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: px-es-data-sc
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 80Gi

Apply the StatefulSet spec for the Elastic search data nodes along with the headless service.

kubectl apply -f es-data-svc.yaml

service "es-data-srv" created

kubectl apply -f es-data-sts.yaml

statefulset "elasticsearch-data" created

kubectl get pods -l "component=elasticsearch, role=data" -w

NAME                   READY     STATUS            RESTARTS   AGE
elasticsearch-data-0   0/1       PodInitializing   0          24s
elasticsearch-data-0   1/1       Running                0          26s
elasticsearch-data-1   0/1       Pending                0          0s
elasticsearch-data-1   0/1       Pending           0          0s
elasticsearch-data-1   0/1       Pending                0          3s
elasticsearch-data-1   0/1       Init:0/1            0          3s
elasticsearch-data-1   0/1       PodInitializing   0          5s
elasticsearch-data-1   1/1       Running                0          51s
elasticsearch-data-2   0/1       Pending                0          0s
elasticsearch-data-2   0/1       Pending                0          0s
elasticsearch-data-2   0/1       Pending                0          3s
elasticsearch-data-2   0/1       Init:0/1            0          3s
elasticsearch-data-2   0/1       PodInitializing   0          5s
elasticsearch-data-2   1/1       Running                0          18s

Cluster state:

kubectl get all

NAME                                            READY     STATUS    RESTARTS   AGE
po/elasticsearch-data-0                         1/1       Running   0          18m
po/elasticsearch-data-1                         1/1       Running   0          18m
po/elasticsearch-data-2                         1/1       Running   0          17m
po/elasticsearch-coordinator-2193029848-5828s   1/1       Running   0          2m
po/elasticsearch-coordinator-2193029848-7xpss   1/1       Running   0          2m
po/elasticsearch-master-0                       1/1       Running   0          12m
po/elasticsearch-master-1                       1/1       Running   0          12m
po/elasticsearch-master-2                       1/1       Running   0          12m

NAME                          CLUSTER-IP     EXTERNAL-IP    PORT(S)          AGE
svc/elasticsearch             10.105.105.41   <pending>     9200:31989/TCP   2m
svc/elasticsearch-discovery   10.106.42.96    <none>        9300/TCP         12m
svc/es-data-srv               None            <none>        9300/TCP         14m
svc/kubernetes                10.96.0.1       <none>        443/TCP          20d

NAME                              DESIRED   CURRENT   AGE
statefulsets/elasticsearch-data   3         3         18m
statefulsets/elasticsearch-master 3         3         18m

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/elasticsearch-coordinator   2         2         2            2           2m

NAME                                      DESIRED   CURRENT   READY     AGE
rs/elasticsearch-coordinator-2193029848   2         2         2         2m

Verify Elastic Search installation.

Verify that Portworx volumes are used for the elasticsearch cluster.
Verify the cluster state by inserting and querying indexes.

Portworx volumes are created with 2 replicas for storing Indexes and Documents for Elasticsearch. This is based on the Storageclass definition.

kubectl get pv

NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY     STATUS    CLAIM                                       STORAGECLASS     REASON    AGE
pvc-60acf83a-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            Delete           Bound     default/px-storage-elasticsearch-data-0     px-es-data-sc               6m
pvc-77b53fcc-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            Delete           Bound     default/px-storage-elasticsearch-master-0   px-es-master-sc             5m
pvc-8a9ae703-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            Delete           Bound     default/px-storage-elasticsearch-master-1   px-es-master-sc             4m
pvc-a48b03bc-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            Delete           Bound     default/px-storage-elasticsearch-data-1     px-es-data-sc               4m
pvc-aa10c4d6-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            Delete           Bound     default/px-storage-elasticsearch-master-2   px-es-master-sc             3m
pvc-b5911226-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            Delete           Bound     default/px-storage-elasticsearch-data-2     px-es-data-sc               3m

kubectl get pvc

NAME                                STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
px-storage-elasticsearch-data-0     Bound     pvc-60acf83a-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-data-sc     8m
px-storage-elasticsearch-data-1     Bound     pvc-a48b03bc-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-data-sc     6m
px-storage-elasticsearch-data-2     Bound     pvc-b5911226-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-data-sc     6m
px-storage-elasticsearch-master-0   Bound     pvc-77b53fcc-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-master-sc   7m
px-storage-elasticsearch-master-1   Bound     pvc-8a9ae703-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-master-sc   7m
px-storage-elasticsearch-master-2   Bound     pvc-aa10c4d6-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-master-sc   6m

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume list

ID                    NAME                                                SIZE      HA   SHARED    ENCRYPTED    IO_PRIORITY    STATUS                          HA-STATE
507832545683763135    pvc-60acf83a-e6b6-11e8-ba45-000c29dcfd58            80 GiB    2    no        no           LOW            up - attached on 70.0.83.216    Up
371131979294695563    pvc-77b53fcc-e6b6-11e8-ba45-000c29dcfd58            80 GiB    2    no        no           LOW            up - attached on 70.0.83.218    Up
87503122675290230     pvc-8a9ae703-e6b6-11e8-ba45-000c29dcfd58            80 GiB    2    no        no           LOW            up - attached on 70.0.83.216    Up
102878661305450280    pvc-a48b03bc-e6b6-11e8-ba45-000c29dcfd58            80 GiB    2    no        no           LOW            up - attached on 70.0.83.218    Up
476683850212083469    pvc-aa10c4d6-e6b6-11e8-ba45-000c29dcfd58            80 GiB    2    no        no           LOW            up - attached on 70.0.83.223    Up
923503818074837680    pvc-b5911226-e6b6-11e8-ba45-000c29dcfd58            80 GiB    2    no        no           LOW            up - attached on 70.0.83.223    Up

kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume list --label pvc=px-storage-elasticsearch-data-0

ID                    NAME                                        SIZE      HA   SHARED    ENCRYPTED     IO_PRIORITY    STATUS                          HA-STATE
507832545683763135    pvc-60acf83a-e6b6-11e8-ba45-000c29dcfd58    80 GiB    2    no        no            LOW            up - attached on 70.0.83.216    Up

kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume inspect 507832545683763135

Volume    :  507832545683763135
    Name                 :  pvc-60acf83a-e6b6-11e8-ba45-000c29dcfd58
    Group                 :  elastic_data_vg
    Size                 :  80 GiB
    Format               :  ext4
    HA                   :  2
    IO Priority          :  LOW
    Creation time        :  Nov 12 20:06:04 UTC 2018
    Shared               :  no
    Status               :  up
    State                :  Attached: disrani-noon-thunder-1 (70.0.83.216)
    Device Path          :  /dev/pxd/pxd507832545683763135
    Labels               :  pvc=px-storage-elasticsearch-data-0,namespace=default
    Reads                :  60
    Reads MS             :  284
    Bytes Read           :  1130496
    Writes               :  2879
    Writes MS            :  315432
    Bytes Written        :  1344454656
    IOs in progress      :  1
    Bytes used           :  1.4 GiB
    Replica sets on nodes:
        Set 0
          Node          : 70.0.83.223 (Pool 1)
          Node          : 70.0.83.216 (Pool 1)
    Replication Status     :  Up
    Volume consumers     :
        - Name           : elasticsearch-data-0 (60adf461-e6b6-11e8-ba45-000c29dcfd58) (Pod)
          Namespace      : default
          Running on     : disrani-noon-thunder-1
          Controlled by  : elasticsearch-data (StatefulSet)

Verify Elastic search cluster state.

Obtain the External IP address from the elasticsearch coordinator service:

kubectl describe svc elasticsearch

Name:            elasticsearch
Namespace:        default
Labels:            component=elasticsearch
            role=coordinator
Annotations:        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"component":"elasticsearch","role":"coordinator"},"name":"elasticsearch","namespa...
Selector:        component=elasticsearch,role=coordinator
Type:              LoadBalancer
IP:                10.105.105.41
Port:                  9200/TCP
NodePort:            32058/TCP
Endpoints:        10.36.0.1:9200,10.47.0.3:9200
Session Affinity:    None
Events:            <none>

kubectl exec elasticsearch-master-0  -- curl "http://elasticsearch.default.svc:9200"

{
  "name" : "elasticsearch-coordinator-2155074821-nxdkt",
  "cluster_name" : "escluster",
  "cluster_uuid" : "zAYA9ERGQgCEclvYHCsOsA",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

kubectl exec elasticsearch-master-0  -- curl "http://elasticsearch.default.svc:9200/_cat/nodes?v"

ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.44.0.2           41          41   0    0.00    0.03     0.08 m         -      elasticsearch-master-0
10.36.0.1           43          18   0    0.07    0.05     0.05 i         -      elasticsearch-coordinator-2155074821-v0w31
10.40.0.2           49          15   0    0.05    0.07     0.11 m         *      elasticsearch-master-1
10.47.0.3           43          20   0    0.13    0.11     0.13 i         -      elasticsearch-coordinator-2155074821-nxdkt
10.47.0.4           42          20   0    0.13    0.11     0.13 d         -      elasticsearch-data-2
10.47.0.2           39          20   0    0.13    0.11     0.13 m         -      elasticsearch-master-2
10.42.0.2           41          13   0    0.00    0.04     0.10 d         -      elasticsearch-data-1
10.40.0.3           42          15   0    0.05    0.07     0.11 d         -      elasticsearch-data-0

kubectl exec elasticsearch-master-0 -- curl -XPUT "http://elasticsearch.default.svc:9200/customer?pretty&pretty"

{
  "acknowledged" : true,
  "shards_acknowledged" : true
}

kubectl exec elasticsearch-master-0 -- curl -XGET "http://elasticsearch.default.svc:9200/_cat/indices?v&pretty"

health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   customer -Cort549Sn6q4gmbwicOMA   5   1          0            0      1.5kb           810b

kubectl exec elasticsearch-master-0 -- curl -XPUT "http://elasticsearch.default.svc:9200/customer/external/1?pretty&pretty" -H 'Content-Type: application/json' -d'
{
"name": "Daenerys Targaryen"
}'

{
  "_index" : "customer",
  "_type" : "external",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "created" : true
}

kubectl exec elasticsearch-master-0 -- curl -XGET "http://elasticsearch.default.svc:9200/customer/external/1?pretty&pretty"

{
  "_index" : "customer",
  "_type" : "external",
  "_id" : "1",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "name" : "Daenerys Targaryen"
  }
}

Install Kibana

Create kibana-svc.yaml with the following content:

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: default
  labels:
    component: kibana
spec:
  selector:
    component: kibana
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 5601
    protocol: TCP

Create kibana-deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: default
  labels:
    component: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
     component: kibana
  template:
    metadata:
      labels:
        component: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:6.3.2
        env:
        - name: "CLUSTER_NAME"
          value: "escluster"
        - name: XPACK_SECURITY_ENABLED
          value: 'false'
        - name: XPACK_GRAPH_ENABLED
          value: 'false'
        - name: XPACK_ML_ENABLED
          value: 'false'
        - name: XPACK_REPORTING_ENABLED
          value: 'false'
        - name: ELASTICSEARCH_URL
          value: 'http://elasticsearch.default.svc:9200'
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 5601
          name: kibana
          protocol: TCP

Deploy the Kibana spec for the Deployment as well as the service:

kubectl apply -f kibana-svc.yaml

service "kibana" created

kubectl describe svc/kibana

Name:     kibana
Namespace:    default
Labels:     component=kibana
Annotations:    kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"component":"kibana"},"name":"kibana","namespace":"default"},"spec":{"ports"...
Selector:   component=kibana
Type:     LoadBalancer
IP:         10.102.83.180
Port:       80/TCP
NodePort:        32645/TCP
Endpoints:        <none>
Session Affinity:    None
Events:            <none>

kubectl apply -f kibana-deployment.yaml

deployment "kibana" created

kubectl get pods -l "component=kibana" -w

NAME                      READY     STATUS    RESTARTS   AGE
kibana-2713637544-4wxsk   1/1       Running   0          12s

kubectl logs po/kibana-2713637544-4wxsk

{"type":"log","@timestamp":"2017-08-23T11:36:19Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}

Verify Kibana Installation

Insert data into Elasticsearch and verify that Kibana is able to search for the data in Elastic Search. This will help create dashboards and visualizations.

Download a sample file named accounts.json:

kubectl exec elasticsearch-master-0 -- curl -XGET "https://docs.portworx.com/samples/k8s/efk/accounts.json?raw=true" -o accounts.json

Insert data into Elasticsearch:

kubectl exec elasticsearch-master-0 -- curl -H "Content-Type:application/json" -XPOST "http://elasticsearch.default.svc:9200/bank/account/_bulk?pretty&refresh" --data-binary "@accounts.json"

To retrieve high-level information about each index in your Elasticsearch cluster, enter the following command:

kubectl exec elasticsearch-master-0 -- curl "http://elasticsearch.default.svc:9200/_cat/indices?v"

health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   bank     7Ll5S-NeSHK3subHKhA7Dg   5   1       1000            0      1.2mb        648.1kb
green  open   .kibana  uJnR9Dp5RdCvAEJ6bg-mEQ   1   1          2            0     10.8kb          5.4kb
green  open   customer -Cort549Sn6q4gmbwicOMA   5   1          1            0      8.2kb          4.1kb

Once you have run the above command you should see bank and customer indices in your elasticsearch cluster. Search for them through your Kibana dashboard.

Bank Index with its documents

bankIndex

Scaling

Portworx runs as a DaemonSet in Kubernetes. Hence when you add a new node to your Kuberentes cluster you do not need to explicitly run Portworx on it.

If you did use the Terraform scripts to create a kubernetes cluster, you would need to update the minion count and apply the changes via Terraform to add a new Node.

Scale your Elastic Search Cluster:

kubectl scale sts elasticsearch-data --replicas=5

statefulset "elasticsearch-data" scaled

kubectl get pods -l "component=elasticsearch, role=data" -w

NAME                   READY     STATUS            RESTARTS   AGE
elasticsearch-data-0   1/1       Running           0          21m
elasticsearch-data-1   1/1       Running           0          21m
elasticsearch-data-2   1/1       Running           0          20m
elasticsearch-data-3   0/1       PodInitializing   0          22s
elasticsearch-data-3   1/1       Running              0          1m
elasticsearch-data-4   0/1       Pending              0          0s
elasticsearch-data-4   0/1       Pending              0          0s
elasticsearch-data-4   0/1       Pending              0          3s
elasticsearch-data-4   0/1       Init:0/1          0          3s
elasticsearch-data-4   0/1       PodInitializing   0          6s
elasticsearch-data-4   1/1       Running              0          9s

kubectl exec elasticsearch-master-0 -- curl "http://elasticsearch.default.svc:9200/_cat/nodes?v"

ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.40.0.2           35          16   0    0.06    0.09     0.13 m         *      elasticsearch-master-0
10.44.0.2           47          42   0    0.00    0.12     0.17 m         -      elasticsearch-master-1
10.44.0.3           49          42   0    0.00    0.12     0.17 d         -      elasticsearch-data-4
10.47.0.4           36          21   0    0.29    0.11     0.12 d         -      elasticsearch-data-2
10.47.0.2           46          21   0    0.29    0.11     0.12 m         -      elasticsearch-master-2
10.42.0.2           41          15   0    0.01    0.15     0.16 d         -      elasticsearch-data-1
10.40.0.3           42          16   0    0.06    0.09     0.13 d         -      elasticsearch-data-0
10.36.0.2           48          20   0    0.01    0.07     0.08 d         -      elasticsearch-data-3
10.47.0.3           51          21   0    0.29    0.11     0.12 i         -      elasticsearch-coordinator-2155074821-nxdkt
10.36.0.1           51          20   0    0.01    0.07     0.08 i         -      elasticsearch-coordinator-2155074821-v0w31

kubectl get pvc

NAME                                STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
px-storage-elasticsearch-data-0     Bound     pvc-60acf83a-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-data-sc     18m
px-storage-elasticsearch-data-1     Bound     pvc-a48b03bc-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-data-sc     16m
px-storage-elasticsearch-data-2     Bound     pvc-b5911226-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-data-sc     15m
px-storage-elasticsearch-data-3     Bound     pvc-98b350a7-e6b8-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-data-sc     2m
px-storage-elasticsearch-data-4     Bound     pvc-b95c996d-e6b8-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-data-sc     1m
px-storage-elasticsearch-master-0   Bound     pvc-77b53fcc-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-master-sc   17m
px-storage-elasticsearch-master-1   Bound     pvc-8a9ae703-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-master-sc   16m
px-storage-elasticsearch-master-2   Bound     pvc-aa10c4d6-e6b6-11e8-ba45-000c29dcfd58   80Gi       RWO            px-es-master-sc   15m

kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume list

ID                    NAME                                        SIZE      HA   SHARED    ENCRYPTED     IO_PRIORITY    STATUS                          HA-STATE
507832545683763135    pvc-60acf83a-e6b6-11e8-ba45-000c29dcfd58    80 GiB    2    no        no            LOW            up - attached on 70.0.83.216    Up
371131979294695563    pvc-77b53fcc-e6b6-11e8-ba45-000c29dcfd58    80 GiB    2    no        no            LOW            up - attached on 70.0.83.218    Up
87503122675290230     pvc-8a9ae703-e6b6-11e8-ba45-000c29dcfd58    80 GiB    2    no        no            LOW            up - attached on 70.0.83.216    Up
1065394113641908183   pvc-98b350a7-e6b8-11e8-ba45-000c29dcfd58    80 GiB    2    no        no            LOW            up - attached on 70.0.83.223    Up
102878661305450280    pvc-a48b03bc-e6b6-11e8-ba45-000c29dcfd58    80 GiB    2    no        no            LOW            up - attached on 70.0.83.218    Up
476683850212083469    pvc-aa10c4d6-e6b6-11e8-ba45-000c29dcfd58    80 GiB    2    no        no            LOW            up - attached on 70.0.83.223    Up
923503818074837680    pvc-b5911226-e6b6-11e8-ba45-000c29dcfd58    80 GiB    2    no        no            LOW            up - attached on 70.0.83.223    Up
620487557616493514    pvc-b95c996d-e6b8-11e8-ba45-000c29dcfd58    80 GiB    2    no        no            LOW            up - attached on 70.0.83.216    Up

Failover

Pod Failover for Elastic search.

Portworx provides durable storage for the Elastic search pods. Cordon a node so that pods do not get scheduled on it, delete a pod manually to simulate a failure scenario and watch the pod get scheduled on another node. However the StatefulSet with Portworx as the volume would reattach the

kubectl get pods -l "component=elasticsearch, role=data"  -o wide

NAME                   READY     STATUS    RESTARTS   AGE       IP          NODE
elasticsearch-data-0   1/1       Running   0          1h        10.40.0.3   pdc3-sm19
elasticsearch-data-1   1/1       Running   0          1h        10.42.0.2   pdc3-sm18
elasticsearch-data-2   1/1       Running   0          1h        10.47.0.4   pdc3-sm16
elasticsearch-data-3   1/1       Running   0          43m       10.39.0.1   70-0-5-129.pools.spcsdns.net
elasticsearch-data-4   1/1       Running   0          1h        10.44.0.3   pdc-dell14

kubectl cordon 70-0-5-129.pools.spcsdns.net

node "70-0-5-129.pools.spcsdns.net" cordoned

kubectl get nodes

NAME                           STATUS                     AGE       VERSION
70-0-5-129.pools.spcsdns.net   Ready,SchedulingDisabled   20d       v1.7.2
pdc-dell14                     Ready                      21d       v1.7.2
pdc-dell15                     Ready                      21d       v1.7.2
pdc-sm13.portcolo.com          Ready                      21d       v1.7.2
pdc3-sm16                      Ready                      21d       v1.7.2
pdc3-sm18                      Ready                      21d       v1.7.2
pdc3-sm19                      Ready                      21d       v1.7.2

Find the docs count on this data node.

kubectl exec elasticsearch-master-0 -- curl "http://elasticsearch.default.svc:9200/_nodes/elasticsearch-data-3/stats/indices"

{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"escluster","nodes":{"Y53C7xqeS-Wi2UHDdE3hgg":{"timestamp":1503479282677,"name":"elasticsearch-data-3","transport_address":"10.39.0.1:9300","host":"10.39.0.1","ip":"10.39.0.1:9300","roles":["data"],"indices":{"docs":{"count":401,"deleted":0}.....

kubectl delete po/elasticsearch-data-3

pod "elasticsearch-data-3" deleted

kubectl get pods -l "component=elasticsearch, role=data"  -o wide -w

NAME                   READY     STATUS        RESTARTS       AGE       IP          NODE
elasticsearch-data-0   1/1       Running               0         1h        10.40.0.3   pdc3-sm19
elasticsearch-data-1   1/1       Running               0         1h        10.42.0.2   pdc3-sm18
elasticsearch-data-2   1/1       Running               0         1h        10.47.0.4   pdc3-sm16
elasticsearch-data-3   1/1       Terminating           0         46m       10.39.0.1   70-0-5-129.pools.spcsdns.net
elasticsearch-data-4   1/1       Running               0         1h        10.44.0.3   pdc-dell14
elasticsearch-data-3   0/1       Terminating           0         46m       <none>      70-0-5-129.pools.spcsdns.net
elasticsearch-data-3   0/1       Terminating           0         46m       <none>      70-0-5-129.pools.spcsdns.net
elasticsearch-data-3   0/1       Terminating           0         46m       <none>      70-0-5-129.pools.spcsdns.net
elasticsearch-data-3   0/1       Pending                  0         0s        <none>        <none>
elasticsearch-data-3   0/1       Pending                  0         0s        <none>        pdc-dell15
elasticsearch-data-3   0/1       Init:0/1              0         0s        <none>        pdc-dell15
elasticsearch-data-3   0/1       PodInitializing    0         3s        10.36.0.2   pdc-dell15
elasticsearch-data-3   1/1       Running               0         6s        10.36.0.2   pdc-dell15

Verify that the same volume has been attached back to the pod which was scheduled post failover.

kubectl exec elasticsearch-master-0 -- curl "http://elasticsearch.default.svc:9200/_nodes/elasticsearch-data-3/stats/indices"

{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"escluster","nodes":{"Y53C7xqeS-Wi2UHDdE3hgg":{"timestamp":1503479456687,"name":"elasticsearch-data-3","transport_address":"10.36.0.2:9300","host":"10.36.0.2","ip":"10.36.0.2:9300","roles":["data"],"indices":{"docs":{"count":401,"deleted":0},

Node Failover

In the case of a StatefulSet if the node is unreachable, which could happen in either of two cases:

The node is down for maintenance
There has been a network partition.

There is no way for kubernetes to know which of the case is it. Hence Kubernetes would not schedule the StatefulSet and the pods running on those nodes would enter the ‘Terminating’ or ‘Unknown’ state after a timeout. If there was a network partition and when the partition heals, kubernetes will complete the deletion of the Pod and remove it from the API server. It would subsequently schedule a new pod to honor the replication requirements mentioned in the Podspec.

For further information : StatefulSet Pod Deletion

Decomissioning a kubernetes node deletes the node object form the APIServer. Before that you would want to decomission your Portworx node from the cluster. Follow the steps mentioned in Decommision a Portworx node Once done, delete the kubernetes node if it requires to be deleted permanently.

Discussion Forum

If you have more questions about this application, please head over to our discussion forum and feel free to ask more questions.