Filebeat configuration for custom Nginx logs for Elasticsearch and Kibana ( FreeBSD 12.0 )

Filebeat is a part of the big elastic ecosystem. It is a tool for getting and moving log data. Log shipper for Logstash, ElasticSearch, Kibana. After installing default Filebeat on a server it reads usually default Nginx configuration. The goal is to make #Filebeat read custom log format:

log_format vcombined '$host '
 '$remote_addr - $remote_user [$time_local] '
 '"$request" $status $body_bytes_sent '
 '"$http_referer" "$http_user_agent"';

Installing beats on a client machine is simple and fast. Beats package will install 3 tools where Filebeat is one that we will be using.

#pkg install beats
#sysrc filebeat_enable="YES"
#sysrc filebeat_config="-path.home /usr/local/share/beats/filebeat/ -path.config /usr/local/etc/"
//copy firebeat.yml to /usr/local/etc/
#filebeat -path.home /usr/local/share/beats/filebeat/ -path.config /usr/local/etc/ -v -e -d "*" modules enable nginx
#filebeat -path.home /usr/local/share/beats/filebeat/ -path.config /usr/local/etc/ -v -e -d "*" setup -e

To make sure it works on a client-side few modifications are required.

Change patterns

/usr/local/share/beats/filebeat/module/nginx/access/ingest#cat default.json 
{
     "description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
     "processors": [
         {
             "grok": {
                 "field": "message",
                 "patterns": [
                     "%{IPORHOST:nginx.access.host} %{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \[%{HTTPDATE:nginx.access.time}\] \"%{GREEDYDATA:nginx.access.info}\" %{NUMBER:nginx.access.response_code:long} %{NUMBER:nginx.access.body_sent.bytes:long} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\""
                 ],
                 "pattern_definitions": {
                     "IP_LIST": "%{IP}(\"?,?\s%{IP})"
                 },
                 "ignore_missing": true
             }
         },
...

Add a new field:

/usr/local/share/beats/filebeat/module/nginx/access/_meta#cat fields.yml
....
- name: host
  type: keyword
  description: >
    The VirtulHost name.
....

Load the ingest pipelines to ElasticSearch (you do this only once):

/usr/local/sbin/filebeat -path.home /usr/local/share/beats/filebeat/ -path.config /usr/local/etc/ -v -e -d "*" setup --pipelines -modules=nginx

You can debug Filebeat by doing:

/usr/local/sbin/filebeat -path.home /usr/local/share/beats/filebeat/ -path.config /usr/local/etc/ -v -e -d "*"

The main config looks like:

/usr/local/etc#cat firebeat.yaml
filebeat.inputs:
- type: log
  enabled: false
  paths:
filebeat.config.modules:
  path: ${path.config}/beats/filebeat.modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
name: wordpress
fields:
    nginx: true
    http_host: wordpress
setup.kibana:
  host: "10.10.10.10:5601"
output.elasticsearch:
  hosts: ["10.10.10.10:9200"]
logging.level: error
logging.to_syslog: true
logging.to_files: false
xpack.monitoring.enabled: false

Module configuration file:

/usr/local/etc/beats/filebeat.modules.d# cat nginx.yml  
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/*.log"]
  error:
    enabled: true

Start service:

#service filebeat start