Zeekurity Zen – Part III: How to Send Zeek Logs to Splunk

Zeekurity Zen Zeries: How To Send Zeek Logs To Splunk

This is part of the Zeekurity Zen Zeries on building a Zeek (formerly Bro) network sensor.

Overview

In our Zeek journey thus far, we’ve:

Now we’ll send our Zeek logs to Splunk, a popular log analysis platform.  This will enable us to quickly search through Zeek’s large dataset and build interesting queries and dashboards.

To do this, we’ll walkthrough these steps:

  1. Configure Zeek to output logs in JSON format for consumption by Splunk.
  2. Create an index in Splunk for Zeek data.
  3. Installing and configuring the Corelight For Splunk app to index and parse Zeek logs in Splunk.
  4. Create a splunk user to run the Splunk Universal Forwarder.
  5. Installing and configuring a Splunk Universal Forwarder to send Zeek logs to a Splunk instance.

Output Zeek logs to JSON

  1. Stop Zeek if it is currently running.
    zeekctl stop
  2. Edit /opt/zeek/share/zeek/site/local.zeek and add the following.
    # Output to JSON
    @load policy/tuning/json-logs.zeek
  3. Restart Zeek and view the logs in /opt/zeek/logs/current to confirm they are now in JSON format.
    zeekctl deploy
    cd /opt/zeek/logs/current
    less conn.log

Create an index in Splunk for Zeek data

It’s best practice to create separate indexes for different types of Splunk data.

  1. Login to your Splunk instance.
  2. In the top right menu navigate to Settings -> Data -> Indexes.
  3. In the Indexes page, click on New Index.
  4. Type “zeek” for Index Name and click Save to create your new index.

[Optional] Install and configure the Corelight For Splunk app

The Corelight For Splunk app is developed by the Corelight team for use with Corelight (enterprise Zeek) and open-source Zeek sensors. We’ll use this app to help parse, index, and visualize Zeek logs. Note that it is completely optional to use this app. You are free to skip this section entirely.

Note that Splunk has also published their own Splunk Add-on for Zeek aka Bro app which helps to ingest Zeek logs but does not feature any sort of dashboards or reports.

  1. Download and install the Corelight for Splunk app onto your Splunk server.  This can either be done within the Splunk server itself or by manually downloading and installing as you would all other Splunk apps.
  2. You can navigate to the app to verify that it is installed correctly.  However, since we have not yet configured our sensor to send data, the dashboards will be blank.
  3. In the top right menu navigate to Settings -> Knowledge -> Event types.
  4. In the App dropdown menu, select Corelight For Splunk and click on corelight_idx.
  5. In the Search string field type index=zeek.  This tells the Corelight for Splunk app to search for data in the “zeek” index we created earlier.

Create the splunk user to run the Splunk Universal Forwarder

  1. Back in the Zeek sensor, create a splunk user and add it to the splunk and zeek groups.
    sudo groupadd splunk
    sudo useradd splunk -g splunk -G zeek
  2. As root/sudo, set a password for the splunk user.
    sudo passwd splunk

Install and configure a Splunk Universal Forwarder

  1. Login to your Splunk account and download the latest Splunk Universal Forwarder.  Once logged in, click “Download Now” for the Linux 64-bit .rpm installer.  Note that Splunk also generates a convenient wget command you can use from the sensor itself once you accept the license agreement. As of this writing, the latest release is version 8.2.0.  If the download URL referenced in the wget command below no longer works, download directly as noted above.
    wget -O splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.2.0&product=universalforwarder&filename=splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64.rpm&wget=true'
  2. Install the forwarder to /opt/splunkforwarder using the rpm command.
    sudo rpm -i splunkforwarder-<…>-linux-2.6-x86_64.rpm
  3. As root/sudo set ownership of /opt/splunkforwarder to the splunk user.
    sudo chown -R splunk:splunk /opt/splunk
  4. Switch to the splunk user.
    su splunk
  5. Start the forwarder to accept the license agreement and create an administrative password.
    cd /opt/splunkforwarder/bin
    ./splunk start --accept-license
  6. Stop the forwarder.
    ./splunk stop
  7. Remove the default data processing limit.  Edit /opt/splunkforwarder/etc/system/local/limits.conf and add the following lines.Note that given the volume of data that Zeek generates, the forwarder may never process all log data if the default limit is not removed.
    [thruput]
    maxKBps = 0 # means unlimited
    
  8. Edit /opt/splunkforwarder/etc/system/local/inputs.conf to monitor desired Zeek logs.  An example inputs.conf is below but may or may not include the logs you wish to ingest. Note that the index and sourcetype fields are leveraging the “zeek” naming scheme to match the “zeek” index we created in Splunk. If you intend to use the Corelight For Splunk app, you’ll want to replace the “zeek” sourcetype prefix with “corelight” as this is what the app is expecting (e.g. replace “zeek_conn” with “corelight_conn”). Modify the index and sourcetype configurations to your needs.
    [default]
    host = sensor
    [monitor:///opt/zeek/logs/current/conn.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_conn
    [monitor:///opt/zeek/logs/current/dns.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_dns
    [monitor:///opt/zeek/logs/current/software.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_software
    [monitor:///opt/zeek/logs/current/smtp.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_smtp
    [monitor:///opt/zeek/logs/current/ssl.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_ssl
    [monitor:///opt/zeek/logs/current/ssh.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_ssh
    [monitor:///opt/zeek/logs/current/x509.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_x509
    [monitor:///opt/zeek/logs/current/ftp.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_ftp
    [monitor:///opt/zeek/logs/current/http.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_http
    [monitor:///opt/zeek/logs/current/rdp.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_rdp
    [monitor:///opt/zeek/logs/current/smb_files.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_smb_files
    [monitor:///opt/zeek/logs/current/smb_mapping.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_smb_mapping
    [monitor:///opt/zeek/logs/current/snmp.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_snmp
    [monitor:///opt/zeek/logs/current/sip.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_sip
    [monitor:///opt/zeek/logs/current/files.log]
    _TCP_ROUTING = *
    index = zeek
    sourcetype = zeek_files
    
  9. Edit /opt/splunkforwarder/etc/system/local/outputs.conf to send data to your Splunk server.  In the sample file below, replace each instance of splunkserver:9997 with your own server name/IP and port number.
    [tcpout]
    defaultGroup = default-autolb-group
    [tcpout:default-autolb-group]
    server = splunkserver:9997
    [tcpout-server://splunkserver:9997]
    
  10. Start the forwarder as the splunk user and confirm it successfully ran.  You can check /opt/splunkforwarder/var/log/splunk/splunkd.log for any issues.
    cd /opt/splunkforwarder/bin
    ./splunk start

Up Next

In Part IV of this series, we’ll introduce threat hunting with Zeek data through actionable Splunk queries.

Related Posts

Elastic Explained: How To Create a Cluster with Docker Compose

Elastic Explained: How To Create a Cluster with Docker Compose

Overview In this guide we'll walkthrough setting up and running an externally accessible three-node Elastic cluster using Docker Compose on Ubuntu Linux 22.04 that's suitable for a home lab or developer / test environment. Our Elastic deployment will include the...

Zeekurity Zen – Part IX: How To Update Zeek

Zeekurity Zen – Part IX: How To Update Zeek

This is part of the Zeekurity Zen Zeries on building a Zeek (formerly Bro) network sensor. Overview In our Zeek journey thus far, we've: Set up Zeek to monitor some network traffic. Used Zeek Package Manager to install packages. Configured Zeek to send logs to Splunk...

Elastic Explained: How To Guides For The Elastic Stack

Elastic Explained: How To Guides For The Elastic Stack

Elastic develops the popular log analytics platform, the Elastic Stack, which supports a variety of search, observability, and security use cases through its many out of the box integrations.  It's a great platform for collecting, analyzing, and visualizing data from...

Transform Your Business & Operate at Peak Efficiency