ELK Installation with ECK Operator
Hi everyone, In this article, I will talk about ELK stack installations using the ECK operator developed by Elastic company for elastic search installations.
Before we start the installations, let’s examine a little bit of ECK Operator.
What is ECK Operator?
Elastic Cloud on Kubernetes (ECK) automates the deployment, provisioning, management, and orchestration of Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, and Elastic Maps Server on Kubernetes based on the operator pattern.
Built using the Kubernetes Operator pattern, ECK installs into your Kubernetes cluster and goes beyond just simplifying the task of deploying Elasticsearch and Kibana on Kubernetes. It focuses on streamlining all those critical operations, such as:
- Managing and monitoring multiple clusters
- Upgrading to new stack versions with ease
- Scaling cluster capacity up and down
- Changing cluster configuration
- Dynamically scaling local storage (includes Elastic Local Volume, a local storage driver)
- Scheduling backups
But ECK is much more than a Kubernetes Operator. In addition to automating all the operational and cluster administration tasks, it focuses on streamlining the entire Elastic Stack on Kubernetes experience. The vision for ECK is to provide an official way to orchestrate Elasticsearch on Kubernetes and provide a SaaS-like experience for Elastic products and solutions on Kubernetes.
Features
- Elasticsearch, Kibana, APM Server, Enterprise Search, and Beats deployments
- TLS Certificates management
- Safe Elasticsearch cluster configuration & topology changes
- Persistent volumes usage
- Custom node configuration and attributes
- Secure settings keystore updates
Supported versions
- Kubernetes 1.20–1.24
- OpenShift 4.6–4.10
- Elasticsearch, Kibana, APM Server: 6.8+, 7.1+, 8+
- Enterprise Search: 7.7+, 8+
- Beats: 7.0+, 8+
- Elastic Agent: 7.10+ (standalone), 7.14+, 8+ (Fleet)
- Elastic Maps Server: 7.11+, 8+
Custom Resource Definitions
- Elastic Agent
- APM Server
- Beats
- Elastic Maps Server
- Elasticsearch Cluster
- Enterprise Search
- Kibana
Let’s get started!
The applications I use while making these installations.
- Azure Kubernetes Service
- Terraform
- Helm
I only added the configurations that I consider important. You can access the whole configuration from this github link.
Installation
Install Operator
Here, the installations were made with terraform scripts using the helm chart prepared for the operator.
However, Installations can also be done by applying manifest files directly.
kubectl create -f https://download.elastic.co/downloads/eck/2.2.0/crds.yamlkubectl apply -f https://download.elastic.co/downloads/eck/2.2.0/operator.yaml
Here are the results.
Install Resources
Resource installations were made by applying manifest files. Here, custom resource definitions created by the operator are used.
ElasticSearch
The storage class to be used for elastic search can be changed if desired.
Elasticsearch is installed with 3 master nodes by default. The number of master replicas can be changed as needed.
However, for elastic, it should be a node machine with at least 4 cores.
Elastic Search Manifest file link
Filebeat
In my construction, logs will be collected by Filebeat, sent to logstash, filtered, and indexed needed in logstash will be made and sent to elasticsearch.
So I made changes in some manifest files.
- I disabled the elastic and kibana reference parameters in the filebeat configuration below. If these are turned on, my logs cannot be sent to logstash.
- I made different namespace restriction definitions with drop_event.when.
- In order to send my logs to logstash, I have given the address of the logstash service that I will create below using the output.logstash parameter.
Kibana
Logstash
The ECK operator does not contain any specific resource definitions for Logstash. Therefore, we will install logstash using the classic deployment object.
The config maps defined in the following yaml file are given as volume in the logstash deployment yaml file.
Apply Manifests
I used the provider named kubectl to be able to apply the custom manifest files prepared with Terraform into a single folder collectively.
All Results
Terraform results
Pod results
Kibana Interface
Username: elastic
Password: The password can be accessed by running the command below.
kubectl get secret -n logging elasticsearch-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode; echoIndexes
Logs
That’s it!
Thanks to this operator that Elastic has released, it only takes a few minutes to set up a fully secure elastic cluster :)
I hope it was an useful article. Have a nice day everybody :) Stay healthy!
If you would like to show your support, tap on the image 👇🎉
Reference Links
