ELK Stack: The Beginner’s Guide
Hello everyone, today I would like to share my experience with ELK Stack. This step-by-step beginner’s guide will allow you to set up your ELK environment and index a sample log file. We will then use the Kibana Dashboard to visualize the indexed record.
Prerequisites
- A system with Centos 8 installed
- Access to a terminal window/command line (Search > Terminal)
- A user account with sudo or root privileges
- Java version 8 or 11 (required for Logstash)
What Does the ELK Stack Stand For?
ELK stands for Elasticsearch, Logstash, and Kibana. They are the three components of the ELK stack.
Elasticsearch (indexes data) — This is the core of the Elastic software. Elasticsearch is a search and analytics engine used to sort through data.
Logstash (collects data) — This package connects to various data sources, caches them, and directs them to storage. As its name suggests, it collects and “stashes” your log files.
Kibana (visualizes data) — Kibana is a graphical tool for visualizing data. Use it to generate charts and graphs to make sense of the raw data in your databases.
Before going ahead with the implementation, let us first have a look at some of the case studies.
Case Studies
NetFlix
Netflix heavily relies on the ELK stack. The company uses the ELK stack to monitor and analyze customer service operations’ security logs. It allows them to index, store, and search documents from more than fifteen clusters which comprise almost 800 nodes.
The famous social media marketing site LinkedIn uses the ELK stack to monitor performance and security. The IT team integrated ELK with Kafka to support its load in real-time. Their ELK operation includes more than 100 clusters across six different data centers.
Tripwire:
Tripwire is a worldwide Security Information Event Management system. The company uses ELK to support information packet log analysis.
Medium:
Medium is a famous blog-publishing platform. Medium uses the ELK stack to debug its production issues. The company also uses ELK to detect DynamoDB hotpots. Moreover, using this stack, the company can support 25 million unique readers and thousands of published posts each week.
Step 1: Install or check OpenJDK 8 Java in your system
We will first install Java 8. To check if you already have java installed or not, use the below command and execute it inside a terminal -
To install Java 8, open a terminal and type the command -
sudo yum install java-1.8.0-openjdk -y
The system will check the repositories and then install java in your system. (Please make sure your internet connection is active.)
Step 2: Add ELK Repository
We will have to add the ELK repository to access Elasticsearch, Logstash, and Kibana.
First, import the Elasticsearch PGP Key. Open a terminal window, then enter the following code:
sudo rpm — import https://artifacts.elastic.co/GPG-KEY-elasticsearch
This command will add the Elasticsearch public signing key to your system. This key will validate the Elasticsearch software when you download it.
Now, you will need to create the repository config file for ELK.
Start by moving into the directory:
cd /etc/yum.repos.d/
Next, create the config file in a text editor of your choice:
sudo vi elasticsearch.repo
Type or copy the following lines:
[elasticstack]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Press Esc, then type “:wq” and hit Enter.
Finally, update your repositories package lists:
yum update
Step 3: Install and Set Up Elasticsearch
The order of installation is essential. Start by installing Elasticsearch.
In a terminal window, type in the command:
sudo yum install elasticsearch -y
This command will scan all your repositories for the Elasticsearch package and install it.
Once the installation finishes, open and edit the configuration file:
sudo vi /etc/elasticsearch/elasticsearch.yml
Scroll down to the section labeled NETWORK. Below that entry, you should see the following lines:
You can adjust the network.host to your server’s IP address or set it to localhost if setting up a single node locally. By default, elastic search bind to port 9200; however, you can adjust the port by changing the value of http.port.
After changing the host and port configuration, you will need to save the file. Press Esc, then type “:wq” and hit Enter.
Now, we will start the elastic search service:
sudo systemctl start elasticsearch
Now, if you want the service to launch at boot, then:
sudo systemctl enable elasticsearch
To test the elastic-search connection:
Step 4: Install and Set Up Kibana
Kibana is a graphical interface for parsing and interpreting log files. Kibana uses the same GPG key as Elasticsearch, so you don’t need to re-import the key. Additionally, the Kibana package is in the same ELK stack repository as Elasticsearch. Hence, there is no need to create another repository configuration file.
To install Kibana, open a terminal window, enter the following:
sudo yum install kibana -y
The system will scan the repositories and then install them.
Now, we will configure the kibana.yml file:
sudo vi /etc/kibana/kibana.yml
You can adjust the server.host to your server’s IP address or set it to localhost if setting up a single node locally. You can also adjust the server.port to change the binding port for Kibana. Also, configure the elastic search.hosts to connect with the Elasticsearch.
Make any other edits as desired, then save the file. Press Esc, then type “:wq” and hit Enter.
Note: Your system may substitute 127.0.0.1:9200 for the localhost:9200 line.
Now, we will start the elastic search service:
sudo systemctl start kibana
Now, if you want the service to launch at boot, then:
sudo systemctl enable kibana
Check if firewalld is enabled in your CentOS system; you need to allow traffic on port 5601. In a terminal window, run the following command:
firewall-cmd — add-port=5601/tcp — permanent
Next, reload the firewalld service:
firewall-cmd — reload
The above action is a prerequisite if you intend to access the Kibana dashboard from external machines.
Now, open a web browser, and enter the address: http://localhost:5601
Step 5: Install and Set Up Logstash
Logstash is a tool that collects data from different sources. The data it collects is parsed by Kibana and stored in Elasticsearch.
Like other parts of the ELK stack, Logstash uses the same Elastic GPG key and repository.
To install Logstash on CentOS 8, in a terminal window, enter the command:
sudo yum install logstash -y
Now, we will create a custom configuration file in the /etc/logstash/conf.d/ directory. Logstash will use this configuration file, parse incoming data and then forward it to Elasticsearch.
First, go to the Logstash configuration directory:
cd /etc/logstash/conf.d/
Create a configuration file:
vi customlogs.conf
Enter the below lines inside the configuration file:
input {
beats {
port => 5044
}
}filter{
}output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "sample-log"
}
}
Press Esc, then type “:wq” and hit Enter.
Below is the complete explanation of each line that we have written inside the file:
input — inside this, we will configure the source of our log/data. For now, I have configured it to parse all the incoming data from file beat on port 5044.
filter — inside this, we ask Logstash to perform some action on data if it meets specific criteria. For now, we are keeping it empty.
output — inside this, we are telling Logstash the location where we have to send the data. We have configured it to send the log to Elasticsearch and store it inside the index sample log.
Note: The index name we are using should be in small letters.
Now, we will start the Logstash service:
sudo systemctl start logstash
Now, if you want the service to launch at boot, then:
sudo systemctl enable logstash
Step 6: Install Filebeat
To simplify logging, install a lightweight module called Filebeat. Filebeat is a shipper for logs that centralizes data streaming.
To install Filebeat, open a terminal window, and run the command:
sudo yum install filebeat -y
Note: Make sure that the Kibana service is up and running during the installation procedure.
Now, we will configure the Filebeat to access a log directory and send it to the Logstash.
Go to the Filebeat directory:
cd /etc/filebeat/
Inside this, you will see the filebeat.yml file. Please open it and look for the Filebeat inputs.
vi filebeat.yml
Now, edit the file with the below information:
- type: log
enabled: true
paths:
- /home/nnewar/logs/*.log [Provide the directory of your log]Your configuration file should look like below:
Now, in the same file, we will configure the output for the Filebeat. By default, you will see Elasticsearch output is enabled. We will comment it out and enable the Logstash output by uncommenting it.
The updated configuration should look like this:
Now, we will start the Filebeat service:
sudo systemctl start filebeat
Now, if you want the service to launch at boot, then:
sudo systemctl enable filebeat
Now, add some log files inside the directory that you have provided inside the filebeat.yml file.
Step 7: Check Logs in Kibana
If everything went well, then you could check the sample-log index inside the Kibana:
Open the Kibana Dashboard: http://localhost:5601
Click on the three lines on the top-left side, then scroll down to Management > Stack Management
Inside the Data > Index Management, you will the index name that we created earlier inside the Logstash.
Step 8: Kibana Dashboard creation
Now, we will create a dashboard for the sample-log index to view the record.
Go to Kibana > Index Patterns present in the same page and click on Create Index Pattern on the top-right side:
To view the records inside a dashboard, we will have to create an Index Pattern for the indexes.
On the next page, you will see the list of all the indexes present currently. For now, we will only see the sample-log index.
Now, enter the index pattern name sample-log inside the box and click on Next step
On the next page, select timestamp from the drop down menu and click on Create Index Pattern
You have successfully created the Index pattern. Now, we will see the records of this index pattern inside the Kibana dashboard.
Click on three lines on the top-left and go to Kibana > Discover
Once you click the Discover tab, you will see all the available records inside the sample-log index pattern.
In the above image, you will see the log information inside the message tag.
Note: You will see the number of records depending on the number of lines present inside the logs. The reason is we have not properly filtered the data during the Logstash configuration. Since this is a beginner guide, I will not mention the filter configuration.
Congratulations, you have successfully configured your first ELK stack environment.
If it was pleasant or helpful to you, please press the 👏 clap button and help others find this story.
Did you know you could give up to 50 claps?