Setup Metricbeat for ELK on Docker

Scenario:

Setup Metricbeat to capture the machine metrics like cpu, memory etc and also metrics for image/containers and visualize on kibana for ELK on Docker.

Solution:

• Create a folder called metricbeat. Inside it add below dockerfile

1
2
3
4
5
ARG ELK_VERSION

FROM docker.elastic.co/beats/metricbeat:${ELK_VERSION}

WORKDIR "/usr/share/metricbeat"

• Create file metricbeat.yml inside config folder with following content.
• Note
• Provide host for the docker.sock

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
metricbeat.modules:
- module: docker
  metricsets:
    - "container"
    - "cpu"
    - "diskio"
    - "event"
    - "healthcheck"
    - "info"
    - "image"
    - "memory"
    - "network"
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elasticsearch:9200"]

• In docker-compose.yml add below
• Note:
• mount var/run/docker.sock
• Set privileged to true and also provide user:root for it to be able to access the logs and push the data to ES

.....
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
  metricbeat:
     build:
       context: ./metricbeat
       args:
          ELK_VERSION: $ELK_VERSION
     volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/local/bin/docker:/usr/bin/docker
      - /sys:/sys
      - type: bind
        source: ./metricbeat/config/metricbeat.yml
        target: /usr/share/metricbeat/metricbeat.yml
        read_only: true
     privileged: true
     user: root
     environment:
       - output.elasticsearch.hosts=["elasticsearch:9200"]
     networks:
       - elk
     depends_on:
       - elasticsearch
     restart: always

• Please note X-Pack has security which on by default and so by default ES is not accessible with out creds. So to skip that still using trial license disable security by adding to  elasticsearch.yml below:
• xpack.security.enabled: false 
• Powershell -> docker-compose up -d --force-recreate --no-deps
• Navigate to Kibana -> Create Index pattern for metricbeat* and then navigate to discover to see the data and visualization.