Zeekurity Zen – Part VIII: How to Send Zeek Logs to Elastic

This is part of the Zeekurity Zen Zeries on building a Zeek (formerly Bro) network sensor.

Overview

In our Zeek journey thus far, we’ve:

In Part III, we sent our Zeek logs to Splunk and unlocked a new level of analysis and visualization capabilities.  But what if we don’t have access to Splunk? What if instead of Splunk, we use Elastic?

For those new to Elastic, the “Elastic Stack” or “ELK Stack” is a popular open source data analytics platform.  Elastic has become increasingly popular not just because it offers a free tier but also through the addition of significant security-focused features and an active user community.

Fortunately for us, Elastic natively supports Zeek logs and is a great way to analyze and visualize Zeek data.

To do this, we’ll walkthrough these steps:

  1. Configure Zeek to output logs in JSON format.
  2. Install and configure Elastic Agent OR Filebeat to consume Zeek logs.  Do not do both.
  3. Write simple Kibana queries.

Note: If you don’t already have an Elastic cluster set up, check out our guide on setting up your own cluster with Docker Compose.  Once you’re ready, come back here.  Also, if you’ve got Microsoft 365, learn how to use Intune to deploy Elastic Agent on your Windows or macOS systems.  With at least one agent configured, you can also collect your Microsoft 365 telemetry and really get a full picture of your environment.

Output Zeek logs to JSON

  1. Stop Zeek if it is currently running.
    zeekctl stop
  2. Edit /opt/zeek/share/zeek/site/local.zeek and add the following.
    # Output to JSON
    @load policy/tuning/json-logs.zeek
  3. Restart Zeek and view the logs in /opt/zeek/logs/current to confirm they are now in JSON format.
    zeekctl deploy
    cd /opt/zeek/logs/current
    less conn.log

Configure Elastic Agent

This guide assumes you have already installed Elastic Agent and a Fleet Server and have created and assigned an Agent policy for your Zeek sensor. If not, refer to Elastic’s documentation and then come back here when you’re done.

  1. In Kibana, navigate to Management -> Integrations.
  2. In the Integrations page, search for “Zeek” and click on the button that appears.
  3. In the Zeek integration page, click on Add Zeek.
  4. In the Add Zeek Integration configuration page, you can change the Integration name if you want, but the remaining defaults should work for most deployments.  The only default you may need to change is the Base Path for where Zeek logs are written to if you customized the location from what was used in the guide (i.e. something other than /opt/zeek/logs/current). At the bottom, for “Where to add this integration?”, you can create a new Agent policy or assign it to your existing Zeek sensor’s Agent policy.  Upon completion click Save and continue.
    
    
  5. After a few minutes, you should start seeing Zeek logs appear in the logs-* data view if you search for event.module:zeek.

Configure Filebeat [Optional]

Note: Elastic Agent is the preferred method for ingesting Zeek logs into Elastic.  If you’re already collecting Zeek logs through Elastic Agent, it is not necessary to also configure Filebeat as this will result in duplicate data.

This guide assumes you have already installed Filebeat. If not, refer to Elastic’s documentation and then come back here when you’re done.

  1. Switch back to your normal user.
    su eric
  2. Stop Filebeat if it is currently running.
    sudo systemctl stop filebeat
  3. Enable Filebeat’s Zeek module.
    sudo filebeat modules enable zeek
  4. Setup Filebeat assets.
    sudo filebeat setup -e
  5. Add Zeek’s log paths (e.g., /opt/zeek/logs/current/http.log) to the Zeek Filebeat configuration file. Assuming you installed Filebeat from the standard Elastic repositories, the configuration file will be in /etc/filebeat/modules.d/zeek.yml. For each log type you want to capture:
    • Set the enabled field to true.
    • Set the var.paths field to the log’s specific path.

    The sample configuration file below enables all log types currently available in the Filebeat Zeek module. Note that while the “signature” log type is listed, it is currently not available and will result in an error if enabled. This Github issue discusses this in further detail.

    # Module: zeek
    # Docs: https://www.elastic.co/guide/en/beats/filebeat/main/filebeat-module-zeek.html
    - module: zeek
      capture_loss:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/capture_loss.log"]
      connection:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/conn.log"]
      dce_rpc:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/dce_rpc.log"]
      dhcp:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/dhcp.log"]
      dnp3:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/dnp3.log"]
      dns:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/dns.log"]
      dpd:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/dpd.log"]
      files:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/files.log"]
      ftp:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/ftp.log"]
      http:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/http.log"]
      intel:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/intel.log"]
      irc:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/irc.log"]
      kerberos:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/kerberos.log"]
      modbus:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/modbus.log"]
      mysql:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/mysql.log"]
      notice:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/notice.log"]
      ntlm:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/ntlm.log"]
      ntp:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/ntp.log"]
      ocsp:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/oscp.log"]
      pe:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/pe.log"]
      radius:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/radius.log"]
      rdp:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/rdp.log"]
      rfb:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/rfb.log"]
      signature:
        enabled: false
        var.paths: ["/opt/zeek/logs/current/signature.log"]
      sip:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/sip.log"]
      smb_cmd:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/smb_cmd.log"]
      smb_files:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/smb_files.log"]
      smb_mapping:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/smb_mapping.log"]
      smtp:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/smtp.log"]
      snmp:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/snmp.log"]
      socks:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/socks.log"]
      ssh:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/ssh.log"]
      ssl:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/ssl.log"]
      stats:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/stats.log"]
      syslog:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/syslog.log"]
      traceroute:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/traceroute.log"]
      tunnel:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/tunnel.log"]
      weird:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/weird.log"]
      x509:
        enabled: true
        var.paths: ["/opt/zeek/logs/current/x509.log"]
        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        #var.paths:
    
  6. Restart Filebeat to apply the configuration and confirm your Zeek logs are now properly ingested into Elasticsearch and available for analysis in Kibana.
    sudo systemctl restart filebeat

Simple Kibana Queries

Once Zeek logs are flowing into Elasticsearch, we can write some simple Kibana queries to analyze our data. Let’s convert some of our previous sample threat hunting queries from Splunk SPL into Elastic KQL. Try taking each of these queries further by creating relevant visualizations using Kibana Lens.

Connections To Destination Ports Above 1024

event.module:zeek AND event.dataset:zeek.connection AND destination.port>1024

Query Responses With NXDOMAIN

event.module:zeek AND event.dataset:zeek.dns AND dns.response_code:NXDOMAIN

Expired Certificates

event.module:zeek AND event.dataset:zeek.x509 AND file.x509.not_after < now

Related Posts

Elastic Explained: How To Create a Cluster with Docker Compose

Elastic Explained: How To Create a Cluster with Docker Compose

Overview In this guide we'll walkthrough setting up and running an externally accessible three-node Elastic cluster using Docker Compose on Ubuntu Linux 22.04 that's suitable for a home lab or developer / test environment. Our Elastic deployment will include the...

Zeekurity Zen – Part IX: How To Update Zeek

Zeekurity Zen – Part IX: How To Update Zeek

This is part of the Zeekurity Zen Zeries on building a Zeek (formerly Bro) network sensor. Overview In our Zeek journey thus far, we've: Set up Zeek to monitor some network traffic. Used Zeek Package Manager to install packages. Configured Zeek to send logs to Splunk...

Elastic Explained: How To Guides For The Elastic Stack

Elastic Explained: How To Guides For The Elastic Stack

Elastic develops the popular log analytics platform, the Elastic Stack, which supports a variety of search, observability, and security use cases through its many out of the box integrations.  It's a great platform for collecting, analyzing, and visualizing data from...

Transform Your Business & Operate at Peak Efficiency