ELK Stack – Filebeat vs Logstash

Filebeat and Logstash are both comprised in ELK Stack, in this blog we will compare both with some use cases. So, you can then take the most appropriate to your case.

Introduction

Are you using ELK for logging? In fact, both Filebeat and Logstash can be used to send logs from a file-based data source to a supported output destination. In most cases, we will be using both in tandem when building a logging pipeline with the ELK Stack because both have a different function, that’s what we will discover together.

So, why compare both products?

Well, people are getting confused by the differences between the two log shippers. With the introduction of Beats, the growth in both their popularity, and the number of use cases, people are inquiring whether the two are complementary or mutually exclusive!

This blog will attempt to shed some light on what makes these two tools both alternatives to each other and complementary at the same time 🙂

What is Logstash?

Logstash was originally developed by Jordan Sissel to handle the streaming of a large amount of log data from multiple sources, and after Sissel joined the Elastic team, Logstash evolved from a standalone tool to an integral part of the ELK Stack (Elasticsearch, Logstash, Kibana).

Logstash advantages

Logstash is an effective centralized logging system, a tool that handle the tasks of pulling and receiving the data from multiple systems, transforming it into a meaningful set of fields and eventually streaming the output to a defined destination for storage.

This sounds cool and enough, so why Filebeat exists? Let’s see the disadvantages 🙂

Logstash disadvantages

Even if Logstash was and still great, there is one outstanding issue, it is performance! Yes, Logstash requires JVM to run, and this dependency coupled with the implementation in Ruby became the root cause of significant memory consumption…

As necessity is the mother of invention… Beats are here 😉

What is Filebeat?

The different Beats are used as lightweight agents installed on the different servers in the infrastructure for shipping logs or metrics. These can be log files (Filebeat), network metrics (Packetbeat), server metrics (Metricbeat), or any other type of data that can be collected by the growing number of Beats being developed by both Elastic and the community.

Filebeat advantages

Filebeat is just a tiny binary with no dependencies, it takes very little resources. That said, you have lots of knobs regarding what it can do. For example, how aggressive it should be in searching for new files to tail and when to close file handles when a file didn’t get changes for a while.

To help you get started, Filebeat comes with modules for specific log types. For example, the Apache module will point Filebeat to default access.log and error.log paths. Personally, I always configure logs entry.

Filebeat disadvantages

Filebeat’s scope is very limited, so you’ll have a problem to solve somewhere else. For example, if you use Logstash down the pipeline, you have about the same performance issue. Because of this, Filebeat’s scope is growing. Initially it could only send logs to Logstash and Elasticsearch, but now it can send to Kafka and Redis. Filebeat can also do some filtering: it can drop events or append metadata to them.

So, what shall I use, Logstash or Filebeat?

Before answering this question, let’s say that Logstash is rarely implemented in modern implementations of the ELK Stack, I do not recommend using Logstash for log processing only, its too heavy and burdensome.

On the other hand, Filebeat is one of the best log file shippers out there today, it’s lightweight, supports SSL and TLS encryption, supports back pressure with a good built-in recovery mechanism, and is extremely reliable. It cannot, however, in most cases, turn your logs into easy-to-analyze structured log messages using filters for log enhancements. That’s the role played by Logstash.

You got it, it depends on your use case 🙂

If you are interested only in the timestamp and message fields to push them to Elasticsearch, I would say, use Filebeat for sure! If you want more, I mean if you want transformation, Logstash could play this role.