In a previous blog, we saw together the difference between Filebeat and Logstash, we said also that both can cooperate together in some cases. In this blog series we will deep into a use case: send logs using filebeat, transform it using Logstash, then send it to Elasticsearch, view it from Kibana.

Let’s start by the first step, which is the installation of all components (Filebeat, Logstash, Elasticsearch, and Kibana). For that I will use an environment with 3 VMs as following:

Filebeat Installation

Let’s install Filebeat on the server we want to monitor (vmelastic1).

To download and install Filebeat, you can use tar package (other options are available here):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[elastic@vmelastic1 app]# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.8.1-linux-x86_64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 41.3M  100 41.3M    0     0   317k      0  0:02:13  0:02:13 --:--:-- 8579k
[elastic@vmelastic1 app]# tar xzvf filebeat-8.8.1-linux-x86_64.tar.gz
filebeat-8.8.1-linux-x86_64/filebeat
filebeat-8.8.1-linux-x86_64/fields.yml
...
[elastic@vmelastic1 app]# ls -rtl filebeat-8.8.1-linux-x86_64
total 140532
-rw-r--r--  1 elastic elastic   2647144 Jun  5 21:56 NOTICE.txt
-rw-r--r--  1 elastic elastic     13675 Jun  5 21:56 LICENSE.txt
drwxr-xr-x  4 elastic elastic        24 Jun  5 22:26 kibana
-rw-------  1 elastic elastic      8622 Jun  5 22:26 filebeat.yml
-rw-r--r--  1 elastic elastic    177676 Jun  5 22:26 filebeat.reference.yml
-rw-r--r--  1 elastic elastic   3745648 Jun  5 22:26 fields.yml
drwxr-xr-x  2 elastic elastic      4096 Jun  5 22:26 modules.d
drwxr-xr-x 71 elastic elastic      4096 Jun  5 22:26 module
-rwxr-xr-x  1 elastic elastic 137281944 Jun  5 22:36 filebeat
-rw-r--r--  1 elastic elastic       809 Jun  5 22:36 README.md

We will configure Filebeat in a next step.

Logstash Installation

Now, we will install Logstash on a separate server (vmelastic2).

To download and install Logstash, you can use tar package (other options are available here):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[elastic@vmelastic2 app]# curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-8.8.1-linux-x86_64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  327M  100  327M    0     0  2399k      0  0:02:19  0:02:19 --:--:-- 25.8M
[elastic@vmelastic2 app]# tar xvf logstash-8.8.1-linux-x86_64.tar.gz
...
logstash-8.8.1/jdk/release
logstash-8.8.1/LICENSE.txt
logstash-8.8.1/logstash-core/lib/logstash/build.rb
[elastic@vmelastic2 app]# ls -rtl logstash-8.8.1
total 664
-rw-r--r-- 1 elastic wheel 606491 Jun  2 19:42 NOTICE.TXT
-rw-r--r-- 1 elastic wheel  13675 Jun  2 19:42 LICENSE.txt
-rw-r--r-- 1 elastic wheel     16 Jun  2 19:42 JDK_VERSION
-rw-r--r-- 1 elastic wheel  30037 Jun  2 19:42 Gemfile.lock
-rw-r--r-- 1 elastic wheel   4101 Jun  2 19:42 Gemfile
drwxr-xr-x 2 elastic wheel      6 Jun  2 19:42 data
-rw-r--r-- 1 elastic wheel   2276 Jun  2 19:42 CONTRIBUTORS
drwxr-xr-x 2 elastic elastic    4096 Jun 27 15:34 bin
drwxr-xr-x 2 elastic elastic     142 Jun 27 15:34 config
drwxr-xr-x 4 elastic elastic      55 Jun 27 15:34 modules
drwxr-xr-x 6 elastic elastic      84 Jun 27 15:34 lib
drwxr-xr-x 3 elastic elastic      30 Jun 27 15:34 tools
drwxr-xr-x 3 elastic elastic      86 Jun 27 15:34 logstash-core-plugin-api
drwxr-xr-x 4 elastic elastic      90 Jun 27 15:34 logstash-core
drwxr-xr-x 4 elastic elastic      33 Jun 27 15:34 vendor
drwxr-xr-x 9 elastic elastic     193 Jun 27 15:34 x-pack
drwxr-xr-x 9 elastic elastic     121 Jun 27 15:35 jdk

Logstash configuration will be done later.

Elasticsearch Installation

It is time to install Elasticsearch on a separate server (vmelastic3).

To download and install Elasticsearch, you can use tar package (other options are available here):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[elastic@vmelastic3 app]# curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.8.1-linux-x86_64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  569M  100  569M    0     0  3927k      0  0:02:28  0:02:28 --:--:-- 27.7M
[elastic@vmelastic3 app]# tar xvf elasticsearch-8.8.1-linux-x86_64.tar.gz
elasticsearch-8.8.1/
elasticsearch-8.8.1/lib/
...
[elastic@vmelastic3 app]# ls -rtl elasticsearch-8.8.1/
total 2224
-rw-r--r--  1 elastic elastic    8106 Jun  5 23:31 README.asciidoc
-rw-r--r--  1 elastic elastic    3860 Jun  5 23:31 LICENSE.txt
drwxr-xr-x  2 elastic elastic       6 Jun  5 23:33 plugins
drwxr-xr-x  2 elastic elastic       6 Jun  5 23:33 logs
-rw-r--r--  1 elastic elastic 2251526 Jun  5 23:33 NOTICE.txt
drwxr-xr-x  5 elastic elastic    4096 Jun  5 23:37 lib
drwxr-xr-x  8 elastic elastic      96 Jun  5 23:37 jdk
drwxr-xr-x  2 elastic elastic    4096 Jun  5 23:37 bin
drwxr-xr-x 74 elastic elastic    4096 Jun  5 23:37 modules
drwxr-xr-x  3 elastic elastic     210 Jun 27 16:05 config

Elasticsearch configuration will be done later.

Kibana Installation

The last one is Kibana, will be installed on the same server as Elasticsearch (vmelastic3).

To download and install Elasticsearch, you can use tar package (other options are available here):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[elastic@vmelastic3 app]# curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-8.8.1-linux-x86_64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  263M  100  263M    0     0  1705k      0  0:02:38  0:02:38 --:--:-- 7537k
kibana-8.8.1/x-pack/plugins/fleet/target/
kibana-8.8.1/x-pack/plugins/fleet/target/agent_versions_list.json
[elastic@vmelastic3 app]# ls -rtl kibana-8.8.1
total 1388
drwxr-xr-x   4 elastic elastic      95 Jun  6 00:19 x-pack
drwxr-xr-x  11 elastic elastic     191 Jun  6 00:19 src
-rw-r--r--   1 elastic elastic    3966 Jun  6 00:19 README.txt
drwxr-xr-x   2 elastic elastic       6 Jun  6 00:19 plugins
drwxr-xr-x   5 elastic elastic      52 Jun  6 00:19 packages
-rw-r--r--   1 elastic elastic     738 Jun  6 00:19 package.json
-rw-r--r--   1 elastic elastic 1370699 Jun  6 00:19 NOTICE.txt
drwxr-xr-x 693 elastic elastic   20480 Jun  6 00:19 node_modules
drwxr-xr-x   2 elastic elastic       6 Jun  6 00:19 logs
-rw-r--r--   1 elastic elastic    3860 Jun  6 00:19 LICENSE.txt
drwxr-xr-x   2 elastic elastic       6 Jun  6 00:19 data
drwxr-xr-x   2 elastic elastic      44 Jun  6 00:19 config
drwxr-xr-x   6 elastic elastic      71 Jun  6 00:19 node
drwxr-xr-x   2 elastic elastic     175 Jun  6 00:19 bin

Next steps

In a next blog, I will configure all these components to push logs from Filebeat to Logstash, make some transformation in Logstash, then send data to Elasticsearch, and see data from Kibana.

Stay connected 😉

Post Views: 2,670
Thumbnail [60x60]
by
David Diab