ELK (Elasticsearch, Logstash, Kibana)

An open-source, free log analysis tool just got better

I am highly interested in analyzing and finding patterns in data. This skill in data analytics is particularly useful when working with log analysis.

What are logs, you may ask?

Logs are records of processes, services, and applications running on your machines.

Why log analysis?

Log analysis helps you identify patterns between processes, allowing you to draw conclusions. It simplifies the task of determining whether a process is malicious or benign.

Why ELK?

ELK is a free, open-source log analysis tool that can be customized to suit your needs. While it is a bit intricate and has a learning curve, who doesn’t love a good free tool?

  1. Elasticsearch: A database that collects and stores all logs from agents in a central place.
  2. Kibana: A tool to visualize log data and customize it according to your needs.
  3. Logstash: A tool that helps define custom filters for logs.

Why containerization?

  1. It is fast, portable, and consumes fewer resources compared to full-fledged applications.
  2. It is easy to scale using container orchestration tools like Kubernetes.
  3. I simply wanted to try it out.

Steps to build a containerized version of the ELK stack:

  1. Download Docker and Docker Compose on your preferred OS (I installed Ubuntu on a virtual machine).
  2. Use the following command to check if Docker is installed properly:
docker run hello-world
  1. Once Docker is installed, create a directory with your preferred name (e.g., elk).
  2. Open the directory and download the .env and docker-compose.yml files.
  3. You will need to specify the version you want to run inside your container using STACK_VERSION = N.N.N.N. Do not use the keyword ‘latest.’
  4. Set an ELASTIC_PASSWORD and KIBANA_PASSWORD.
  5. Edit the docker-compose.yml file to incorporate Logstash. Paste the following code into docker-compose.yml (note: it should be inside the services block and properly aligned):
  # This is an IaC (Infrastructure as Code) template so that this setup can be easily replicated across any system.
  logstash:
    depends_on:
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/logstash/logstash:${VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    volumes:
      - logstashdata01:/usr/share/logstash/data
      - certs:/usr/share/logstash/certs
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
    environment:
      - NODE_NAME=:"logstash"
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
    command: logstash -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044/UDP"
    mem_limit: ${MEM_LIMIT}
  1. Our Logstash service depends on Elasticsearch (es01) and Kibana being healthy. It will start only when es01 and Kibana are online and healthy.
  2. In the volumes section, we mount our local directories in Docker for persistent data. We use certificates generated during the setup to authenticate with es01 and Kibana and initialize Logstash with a local config file.
  3. Specify the username for es01. You can change the username and assign roles once the instances are created.
  4. Declare a volume inside the volumes section:
# When building the Docker container, 
#it will create the following volumes for each service.
volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  kibanadata:
    driver: local
  logstashdata01: 
    driver: local
  1. After this, create a file called logstash.conf, and paste the following into that file. Customize it according to your needs:
input {
    udp{
        host => "0.0.0.0"
        port => 5044 # Ingest logs on this port
    }
}

filter:{}

output{
    elasticsearch{
        index => "logstash-%{+YYYY.MM.dd}"
        hosts => ["https://es01:9200"]
        user => "elastic"
        password => "BADPASSWORD" // Replace with your chosen password
        ssl_enabled => true
        cacert => "usr/share/logstash/certs/ca/ca.crt"
    }
}
  1. As of now, we are not applying any filters to the logs.
  2. Perform a test boot-up using the following command:
docker compose up -d # If the docker-compose.yml file is in the same directory
docker compose -f /path/to/directory -d # If the docker-compose.yml file is in another directory
  1. In the next article, we will perform log analysis.
  2. I have attached my .env, docker-compose.yml, and logstash.conf files here for reference if needed.

Thank you for your patience! Feel free to contact me if you get stuck or if I have explained something incorrectly.

That’s it for now!