Filebeat on OpenBSD 6.2

How to install Filebeat on OpenBSD 6.2, including setup and configuration.

I recently installed a new of OpenBSD 6.2 server. As part of that install, I needed to install Filebeat to forward logs to an ELK server (Elasticsearch, Logstash and Kibana).

The process to install and configure are much simpler then previous OpenBSD versions (as covered here) because Go 1.7 is now a simple binary package install.

The following steps assumes you have installed OpenBSD 6.2 and will cover dependent package and Filebeat installation, and configuration of Filebeat.

Install Go, git and gmake

First, configure `pkg_add` to use the closest binary packages mirror. Select the closest / fastest mirror here:

https://www.openbsd.org/ftp.html

Next edit `/etc/installurl` to set the mirror:

doas vi /etc/installurl

https://ftp.openbsd.org/pub/OpenBSD

Then install Go, git and gmake:

doas pkg_add git gmake go bash

Install Filebeat

Set up the Go build environment and get the Filebeat source:

mkdir ~/go
export GOPATH=~/go
mkdir -p $GOPATH/src/github.com/elastic
cd $GOPATH/src/github.com/elastic
git clone https://github.com/elastic/beats.git

List the branches and switch to the appropriate branch, in this example release v6.2.1:

cd beats
git fetch --all --tags --prune
git checkout tags/v6.2.1
git branch -a

Build Filebeat:

cd filebeat
go get
gmake

Install Filebeat and set permissions:

doas cp -R $GOPATH/bin/filebeat /usr/sbin/
doas chmod 555 /usr/sbin/filebeat
doas chown root.bin /usr/sbin/filebeat

Copy Filebeat config files:

doas mkdir /etc/filebeat
doas cp $GOPATH/src/github.com/elastic/beats/filebeat/filebeat.yml /etc/filebeat/

Copy TLS Cert

If you are using TLS (https) to secure the connection, copy the cert from the ELK server to the OpenBSD server running Filebeat:

scp /etc/logstash/logstash.crt USERNAME@SERVERNAME:/home/USERNAME/logstash.crt

On the OpenBSD server running Filebeat:

doas mv /home/USERNAME/logstash.crt /etc/filebeat
doas chown root.wheel /etc/filebeat/logstash.crt 

Configure Filebeat

Filebeat can be configured to log to Elasticsearch or Logstash, in this example we are logging to Logstash.

Below is an example `filebeat.yml`, please note that this will need to be customized to include what you want to forward. Sections that are not changed are omitted with `[…]`:

sudo vi /etc/filebeat/filebeat.yml

[...]
- type: log

  # Change to true to enable this prospector configuration.
  # CHANGED
  #enabled: false
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    # BEGIN CHANGED
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - /var/log/daemon
    - /var/log/messages
    - /var/log/authlog
    # END CHANGED

[...] 

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1
  
  # BEGIN ADDED
  fields:
    type: syslog
    tag: SERVERNAME_HERE
  # END ADDED
     
[...]  

#-------------------------- Elasticsearch output ------------------------------
# CHANGED
#output.elasticsearch:
  # Array of hosts to connect to.
  # CHANGED
  #hosts: ["localhost:9200"]

[...]

#----------------------------- Logstash output --------------------------------
# CHANGED
output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]
  
  # ADDED
  hosts: ["192.168.2.10:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # ADDED
  ssl.certificate_authorities: ["/etc/filebeat/logstash.crt"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

[...]

Test config file:

doas /etc/filebeat/filebeat.yml test config

Config OK

For details on further configuration please see:

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration.html

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html

For details on setting up TLS/SSL please see:

https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html

Debugging Filebeat

If you having issues with the configuration, enable debugging by adding the following to the end of `filebeat.yml`:

doas vi /etc/filebeat/filebeat.yml

[...]

# BEGIN ADDED
logging:
  level: warning
  to_files: true
  files:
    path: /var/log/filebeat
    name: beat.log
    keepfiles: 7
    rotateeverybytes: 10485760 # 10 MB
  level: debug 
  selectors: ["*"]
# END ADDED

This will log too:

/var/log/filebeat/beat.log

Filebeat rc File

Create the filebeat rc file to automatically start at boot:

doas vi /etc/rc.d/filebeat

#!/bin/sh
#

daemon="/usr/sbin/filebeat"
daemon_flags="-c /etc/filebeat/filebeat.yml"

. /etc/rc.d/rc.subr

rc_bg=YES
rc_reload=NO

rc_pre() {
	install -d -o root -m 0700 /var/db/filebeat
}

rc_start() {
	${rcexec} "${daemon} -s /var/run/fail2ban.sock start ${daemon_flags} ${_bg}"
}

rc_check() {
	pgrep -T "${daemon_rtable}" -q -xf "${pexp}"
}

rc_stop() {
        ${rcexec} "${daemon} -s /var/run/fail2ban.sock stop"
}

rc_cmd $1

Set permissions:

doas chmod 555 /etc/rc.d/filebeat

Add a reference to the rc file to `rc.conf.local` so that it starts up on boot:

doas vi /etc/rc.conf.local

# ADDED filebeat
pkg_scripts="filebeat"

Reference for the rc file:

http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/ports/sysutils/beats/filebeat/pkg/filebeat.rc?rev=1.1.1.1&content-type=text/plain

Manage Filebeat

Use rc script to manage:

sudo /etc/rc.d/filebeat check
sudo /etc/rc.d/filebeat start
sudo /etc/rc.d/filebeat stop

Conclusion

Thats it, hope that you have found this useful, having the Go version in the ports makes this trivial compared to previous versions of OpenBSD.

Tagged OpenBSD | Posted on Sunday, March 25th, 2018 | Comments Off on Filebeat on OpenBSD 6.2