Filebeat on OpenBSD 6.0

In an effort to improve monitoring, I setup an ELK (Elasticsearch, Logstash, Kibana) server and setup my different servers to forward their logs. Filebeat is typically installed on the servers to do the forwarding and normally this installation is pretty straight forward.

However, Filebeat is dependent on Go 1.7 and OpenBSD 6.0 only provides 1.6 as a binary package.

The following steps will setup an OpenBSD ports build machine, update the ports to current, build the required packages; then copy the packages to the target OpenBSD server, install the packages, install and configure Filebeat.

There is a more current version of this document.
For OpenBSD 6.2 please see:
Filebeat on OpenBSD 6.2

Setup OpenBSD Ports Build Machine

Building from the ports system isn’t hard, but the partition layout and not wanting to have the ports tree on a production server resulted in my decision to create a VM just to build ports.

The default partition layout that the OpenBSD 6.0 installer wants to use does not provide enough disk space to build Go and it’s dependencies.

During the install, when prompted to partition the drive, I removed all partitions except swap and then added all the space back as root ‘/’.

Note, this is not a recommended partition strategy and is not something that should be run as a production server, but as a short lived build server, it is a reasonable shortcut.

After installation, I installed `sudo` (like an animal) and configured the `/etc/sudoers` file to allow the wheel group to sudo:

pkg_add sudo
# Uncomment to allow people in group wheel to run all commands
# and set environment variables.
# CHANGED
#%wheel  ALL=(ALL) SETENV: ALL
%wheel  ALL=(ALL) SETENV: ALL

The final step in setting up the ports build machine is to update:

export PKG_PATH="http://ftp.openbsd.org/pub/OpenBSD/$(uname -r)/packages/$(arch -s)/"
sudo pkg_add -uvi

Fetching Ports Tree

With an updated ports build machine, fetch the ports tree and update it from CVS.

Fetch the `ports.tar.gz` from the nearest mirror or ftp.openbsd.org:

cd /tmp
ftp https://ftp.openbsd.org/pub/OpenBSD/$(uname -r)/ports.tar.gz

Validate the download:

cd /tmp
ftp https://ftp.openbsd.org/pub/OpenBSD/$(uname -r)/SHA256.sig
signify -Cp /etc/signify/openbsd-$(uname -r | cut -c 1,3)-base.pub -x SHA256.sig ports.tar.gz

You are looking for a response like:

Signature Verified
ports.tar.gz: OK

Next, uncompress the archive to `/usr`:

cd /usr
sudo tar xzf /tmp/ports.tar.gz

Updating From CVS

This next step is potentially dangerous to the stability of your system. In fact, the binary packages created may not work on OpenBSD 6.0 and I may just have gotten lucky building a working Go 1.7 that works on OpenBSD 6.0.

We are building Go 1.7 from current to install on a stable system.

If that hasn’t scared you off, please read https://www.openbsd.org/faq/faq5.html to understand the release, stable and current flavors.

Looking at the commits for Go 1.7, we want to update the ports tree to current:

cd /usr
sudo cvs -d anoncvs@anoncvs.ca.openbsd.org:/cvs get -P ports

If the update is successful, build Go and it’s dependencies.

At this point it might be worthwhile installing `screen` so that if the compile takes long, you can disconnect from the terminal:

export PKG_PATH="http://mirror/pub/OpenBSD/6.0/packages/amd64/"
sudo pkg_add screen
screen

Finally, build:

cd /usr/ports/lang/go
sudo make install

Once complete the binary package files should be in:

ls /usr/ports/packages/amd64/all                                                                                                                                                       
bash-4.3.48.tgz
gettext-0.19.8.1.tgz
go-1.7p0.tgz
go-bootstrap-1.4.3p2.tgz
libiconv-1.14p3.tgz
xz-5.2.2p0.tgz

Copy all the binary package files to the target OpenBSD server and move them to `/usr/ports/packages/amd64/all`.

On the target OpenBSD server, install the binary packages:

cd /usr/ports/packages/amd64/all
sudo pkg_add *

The ports build machine can now be shutdown.

Install Filebeat

On the target OpenBSD server, install a few dependencies:

export PKG_PATH="http://ftp3.usa.openbsd.org/pub/OpenBSD/$(uname -r)/packages/$(arch -s)/"
sudo pkg_add git gmake

Set up the Go build environment and get the Filebeat source:

mkdir ~/go
export GOPATH=~/go
mkdir -p $GOPATH/src/github.com/elastic
cd $GOPATH/src/github.com/elastic
git clone https://github.com/elastic/beats.git

Switch to the 5.0 release branch:

cd beats
git checkout -b remotes/origin/5.0
git fetch
git checkout 5.0

Build Filebeat:

cd filebeat
go get
gmake

Install Filebeat and set permissions:

sudo cp -R $GOPATH/bin/filebeat /usr/sbin/
sudo chmod 555 /usr/sbin/filebeat
sudo chown root.bin /usr/sbin/filebeat

Copy Filebeat config files:

sudo mkdir /etc/filebeat
sudo cp /home/gturner/go/src/github.com/elastic/beats/filebeat/filebeat.yml /etc/filebeat/
sudo cp /home/gturner/go/src/github.com/elastic/beats/filebeat/filebeat.template.json /etc/filebeat/
sudo cp /home/gturner/go/src/github.com/elastic/beats/filebeat/filebeat.template-es2x.json /etc/filebeat/

Create Filebeat rc file:

sudo vi /etc/rc.d/filebeat
#!/bin/sh
#

daemon="/usr/sbin/filebeat"
daemon_flags="-c /etc/filebeat/filebeat.yml"

. /etc/rc.d/rc.subr

rc_bg=YES
rc_reload=NO

rc_pre() {
	install -d -o root -m 0700 /var/db/filebeat
}

rc_start() {
	${rcexec} "${daemon} ${daemon_flags} ${_bg}"
}

rc_check() {
	pgrep -T "${daemon_rtable}" -q -xf "${pexp}"
}

rc_stop() {
	pkill -T "${daemon_rtable}" -xf "${pexp}"
}

rc_cmd $1

Set permissions:

sudo chmod 555 /etc/rc.d/filebeat

Reference for the rc file:

http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/ports/sysutils/beats/filebeat/pkg/filebeat.rc?rev=1.1.1.1&content-type=text/plain

Add a reference to the rc file to `rc.conf.local` so that it starts up on boot:

sudo vi /etc/rc.conf.local
# ADDED filebeat
pkg_scripts="filebeat"

Configure Filebeat

Filebeat can be configured to log to Elasticsearch or Logstash. Logstash is probably the better option, especially when using TLS/SSL.

Below is an example `filebeat.yml`, please note that this will need to be customized to include what you want to forward:

sudo vi /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/daemon
    - /var/log/messages
    - /var/log/authlog
  document_type: log
  fields:
    type: syslog
    tag: access2
output.logstash:
  hosts: ["YOUR_LOGSTASH_SERVER:5044"]
  protocol: https
  tls:
      certificate_authorities: ["/etc/filebeat/logstash.crt"]
filebeat.registry_file: /var/db/filebeat/.filebeat
path.data: /var/db/filebeat
path.logs: /var/log
path.home: /etc/filebeat

For details on setting up TLS/SSL please see:

https://www.elastic.co/guide/en/beats/filebeat/5.0/configuring-ssl-logstash.html

Manage Filebeat

Use rc script to manage:

sudo /etc/rc.d/filebeat check
sudo /etc/rc.d/filebeat start
sudo /etc/rc.d/filebeat stop

Filebeat logs to:

tail -f /var/log/filebeat

Conclusion

Thats it, hope that you have found this useful, when OpenBSD 6.1 comes out it will likely include Go 1.7 so these instructions will likely not be necessary.

Comments are closed.