Docker is everywhere and Logstash is not going anywhere. So why not try running Logstash on Docker. I found that while Elasticsearch has done a good job of describing the details, the learning curve is still a bit steep for a person new to Docker. Hence this post.
WHY
As usual I will discuss the why first. Logstash remains on of the most competent free open source ETL tool out there. It happily runs on low spec machines munching data. Many people like to run Logstash in AWS EC2 or their own on-prem server. In both the cases there is the issue of server security patches. This is a bigger issue in AWS (or any cloud platform for that matter) where provider keeps releasing security patches for the OS. Security will be on your heels to keep applying the patches. This is where running Logstash on Docker becomes viable. It allows you a way out.
HOW
It is simple. Run Logstash in AWS using Amazon ECS (Elastic Container Service). You have two options. One is to create EC2 instances and let ECS handle them. Other option is using ECS with Fargate. Fargate is serverless. So you will not be bothered with creation of EC2, scaling up or down and so on.
PRO TIP: When working in cloud always calculate your costs. With Logstash it is a very tricky calculation. Constant stream of incoming data is best handled with a server to fix your costs. EC2 can be a very cheap and compelling option for simple cases. In that case ECS with a bunch of low end EC2 instances is good enough. But if you are going to be dealing with huge amount of data with varying volumes and velocity over a given day, Fargate can actually turn out to be cheaper. So do the Maths.
To keep it brief, I will split this in two parts. First part I will show you how to create a image with needed config changes so that you can actually run Logstash on Docker. Next part will be where I will integrate this with AWS and deploy Logstash Docker image on AWS Fargate.
But first install Docker on your machine. You get installers for both Windows as well as Linux.
These two commands should work and print out the respective versions.
docker --version docker compose versionOnce done we are ready.
Your Logstash on Docker journey starts with a file. Dockerfile
Yes there are no extensions to this file and make sure the you spell it with a capital D.
FROM docker.elastic.co/logstash/logstash:8.17.2 # Remove default pipeline configuration file RUN rm -f /usr/share/logstash/pipeline/logstash.conf # Copy custom config file for logstash. Input and output listed in it. COPY http_to_s3.conf /usr/share/logstash/pipeline/http_to_s3.conf # Copy custom logstash.yml COPY logstash.yml /usr/share/logstash/config/logstash.yml # Copy custom pipelines.yml COPY pipelines.yml /usr/share/logstash/config/pipelines.yml
The first line is where we use the docker image from Elasticsearch as the foundation. Idea is to use the image and make the needed tweaks to make it run the way we want. I am using 8.17.2 which is latest at the time of writing. Second line is where I get rid of the configuration file which the image comes with. Instead I will replace it with a file called http_to_s3.conf
input
{
http
{
port => 5044
codec => "json"
}
}
output
{
stdout { codec => rubydebug }
# s3
# {
# region => "us-east-1"
# bucket => "logstash"
# prefix => "data/%{+YYYY}/%{+MM}/%{+dd}/"
# codec => "json"
# canned_acl => "private"
# }
}
Do not worry. I will not be doing any s3 related stuff here. Its all commented out. So the file essentially uses http_input plugin of Logstash. It is rather easy to test using curl hence the choice. The output section essentially has only one statement. It prints out whatever is sent to Logstash.
You have to tell Logstash to use this file in the pipeline it runs. Hence the need for another file called pipelines.yml. I copy it into the image using COPY command in Dockerfile.
# This file is where you define your pipelines. You can define multiple. # For more information on multiple pipelines, see the documentation: # https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html #- pipeline.id: main # path.config: "/usr/share/logstash/pipeline" - pipeline.id: http_to_s3 path.config: "/usr/share/logstash/pipeline/http_to_s3.conf"
This file tells logstash where to pick up the configuration file from.
Lastly some housekeeping and that is the job of the third file logstash.yml.
http.host: "0.0.0.0" pipeline.ordered: false queue.type: persisted queue.max_bytes: 200mb dead_letter_queue.enable: true dead_letter_queue.max_bytes: 50mb
This one is rather small. I am just enabling persistence in Logstash so that the chance of data loss in case of network outage is less.
Keep all the four files in same directory.
I am using a Linux machine. For Windows the command syntax might change slightly. Run this command:
docker image build -t logstash-8.7.12:0.1 .
This should run successfully. Sample output
[+] Building 5.1s (10/10) FINISHED docker:default => [internal] load build definition from Dockerfile 0.2s => => transferring dockerfile: 558B 0.0s => [internal] load metadata for docker.elastic.co/logstash/logstash:8.17.2 2.5s => [internal] load .dockerignore 0.1s => => transferring context: 2B 0.0s => [1/5] FROM docker.elastic.co/logstash/logstash:8.17.2@sha256:7b86ddaf03b6ac359e4f430ed9f54dfcb22570aa1ec0da155166c9377fc15dd2 0.2s => => resolve docker.elastic.co/logstash/logstash:8.17.2@sha256:7b86ddaf03b6ac359e4f430ed9f54dfcb22570aa1ec0da155166c9377fc15dd2 0.2s => [internal] load build context 0.3s => => transferring context: 1.23kB 0.0s => CACHED [2/5] RUN rm -f /usr/share/logstash/pipeline/logstash.conf 0.0s => [3/5] COPY http_to_s3.conf /usr/share/logstash/pipeline/http_to_s3.conf 0.5s => [4/5] COPY logstash.yml /usr/share/logstash/config/logstash.yml 0.3s => [5/5] COPY pipelines.yml /usr/share/logstash/config/pipelines.yml 0.5s => exporting to image 0.3s => => exporting layers 0.2s => => writing image sha256:7e6740550caadb8304ca91b0d26a0e648952c3f4bd7d8e3a8fcaf5b032042dc5 0.0s => => naming to docker.io/library/logstash-8.7.12:0.1 0.0sWe are not doing much here. Just changing a few config files.
Once that is done it’s time to run Logstash on Docker.
docker run --rm -p 5044:5044 logstash-8.7.12:0.1
You should not be seeing any error. The -p maps port 5044 to 5044. Docker has created a port 5044 as requested and any traffic sent to the port 5044 of your machine is sent to port 5044 created by Docker. This is the port to which Logstash is listening to. I am pasting last few lines of the sample output to show how it looks like.
[2025-03-01T03:52:39,903][INFO ][logstash.javapipeline ][http_to_s3] Pipeline Java execution initialization time {"seconds"=>1.05}
[2025-03-01T03:52:39,939][INFO ][logstash.codecs.json ][http_to_s3] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2025-03-01T03:52:40,166][INFO ][logstash.javapipeline ][http_to_s3] Pipeline started {"pipeline.id"=>"http_to_s3"}
[2025-03-01T03:52:40,168][INFO ][logstash.inputs.http ][http_to_s3][b79b012f99a194156bbe0dd9aea74c069bc1afcdea254d88f84a38b690825a3d] Starting http input listener {:address=>"0.0.0.0:5044", :ssl_enabled=>false}
[2025-03-01T03:52:40,186][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:http_to_s3], :non_running_pipelines=>[]}
Once Logstash is running its time to test it by sending it a message. It should appear in the output.
Open another terminal and send it a message using curl. To run the command in windows you might have to change the quotation marks.
curl -X POST "http://localhost:5044" -H "Content-Type: application/json" -d '{"message": "Hello Logstash"}'
Here is the output you will see in the other terminal where Logstash is running.
{
"http" => {
"method" => "POST",
"request" => {
"body" => {
"bytes" => "29"
},
"mime_type" => "application/json"
},
"version" => "HTTP/1.1"
},
"message" => "Hello Logstash",
"user_agent" => {
"original" => "curl/8.9.1"
},
"@timestamp" => 2025-03-01T03:56:09.913217644Z,
"event" => {
"original" => "{\"message\": \"Hello Logstash\"}"
},
"url" => {
"port" => 5044,
"path" => "/",
"domain" => "localhost"
},
"@version" => "1",
"host" => {
"ip" => "172.17.0.1"
}
}
And there you go. A simple quick and no nonsense way to get started with Logstash on Docker !
Cntrl+C will kill the Logstash and also remove the container which Docker created to run it.
You can remove the image to save on space using the command (this is for Linux)
docker rmi -f $(docker images -q)I have kept this one short and sweet. In case you have any doubts regarding any steps put in a comment and I will try to respond.