Setup Filetbeat for ELK on Docker

Scenario:

Setup FileBeat to process the logs and visualize on kibana for ELK on Docker.

Solution:

• Create a folder called filebeat. Inside it add below dockerfile

1
2
3
4
5
ARG ELK_VERSION

FROM docker.elastic.co/beats/filebeat:${ELK_VERSION}

WORKDIR "/usr/share/filebeat"

• Create file filebeat.yml inside config folder with following content.
• Note
• This monitors logs for all running containers
• Its setup index filebeat-* format on ES and also template for it with dashboard on Kibana.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
filebeat.inputs:
- type: log
  paths:
    - /var/lib/docker/containers/*/*.log

setup.template.name: "filebeat-"
setup.template.pattern: "filebeat-*"
setup.dashboards.enabled: true

output.elasticsearch:
   hosts: 'elasticsearch:9200'
   index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

setup.kibana:
  host: "kibana:5601"

• In docker-compose.yml add below
• Note:
• mount lib/docker/containers
• Set privileged to true and also provide user:root for it to be able to access the logs and push the data to ES

.....
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
filebeat:
     build:
       context: ./filebeat
       args:
          ELK_VERSION: $ELK_VERSION
     volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers
      - /sys:/sys
      - type: bind
        source: ./filebeat/config/filebeat.yml
        target: /usr/share/filebeat/filebeat.yml
        read_only: true
     privileged: true
     user: root
     environment:
       - output.elasticsearch.hosts=["elasticsearch:9200"]
     networks:
       - elk
     depends_on:
       - elasticsearch
       - kibana
     restart: always

• Please note X-Pack has security which on by default and so by default ES is not accessible with out creds. So to skip that still using trial license disable security by adding to  elasticsearch.yml below:
• xpack.security.enabled: false 
• Powershell -> docker-compose up -d --force-recreate --no-deps
• Navigate to Kibana -> Discover to see the data and visualization.