This is part of the Zeekurity Zen Zeries on building a Zeek (formerly Bro) network sensor.
Overview
In our Zeek journey thus far, we’ve:
- Set up Zeek to monitor some network traffic.
- Used Zeek Package Manager to install packages.
- Configured Zeek to send logs to Splunk for analysis.
- Uncovered notable events through basic threat hunting.
- Leveraged threat intelligence to elevate our threat hunting game.
- Detected malicious files by hash and extracted commonly exploited file types.
We’ve now experienced Zeek’s powerful analysis capabilities and the deep visibility it enables into all types of network traffic. However, as the internet increasingly defaults to encrypted channels, we might ask ourselves: How do we analyze encrypted traffic like SSL/TLS/SSH for suspicious activity? Can Zeek decrypt encrypted traffic?
Though Zeek cannot decrypt encrypted traffic, it fortunately offers a variety of clever methods for analyzing and understanding encrypted traffic. This includes details on TLS handshakes, X.509 certificates, and SSH connections. You’ll be pleasantly surprised by just how much you can learn about encrypted traffic without actually decrypting it.
To do this, we’ll walkthrough these steps:
- Enable X.509 logging of the full certificate chain.
- Understand ssl.log and x509.log.
- Fingerprint SSL/TLS with JA3.
- Analyze decrypted HTTP/2 traffic.
- Understand ssh.log.
- Fingerprint SSH with HASSH.
Enable X.509 Logging of The Full Certificate Chain
- By default, Zeek only logs the X.509 certificate of the actual host and will exclude any intermediary certificates. To enable logging of the full certificate chain, we need to edit /opt/zeek/share/zeek/site/local.zeek and comment out the following lines by adding a # symbol in front. Then save the file.
# This script prevents the logging of SSL CA certificates in x509.log # @load protocols/ssl/log-hostcerts-only - As the zeek user, stop zeek.
zeekctl stop - As the zeek user, apply the new setting and start zeek.
zeekctl deploy
Understand ssl.log and x509.log
As we noted earlier, Zeek cannot directly decrypt encrypted SSL/TLS traffic, but that doesn’t prevent it from observing the SSL/TLS handshakes or capturing certificate details in cleartext.
- Take a look at your own ssl.log and note the encrypted SSL/TLS connections and details that are captured. Below is a sample ssl.log file in JSON format.
{ "ts": 1600552002.022206, "uid": "CY188jv7sKlzAMNNc", "id.orig_h": "10.0.1.8", "id.orig_p": 54458, "id.resp_h": "104.244.42.1", "id.resp_p": 443, "version": "TLSv12", "cipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "curve": "secp256r1", "server_name": "twitter.com", "resumed": false, "next_protocol": "h2", "established": true, "cert_chain_fuids": [ "FzHo3q14WMvHTjCc79", "F5woTi2O9Ea05rRxhj" ], "client_cert_chain_fuids": [], "subject": "CN=twitter.com,OU=atla,O=Twitter\\, Inc.,L=San Francisco,ST=California,C=US", "issuer": "CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US", "validation_status": "ok", "ja3": "b32309a26951912be7dba376398abc3b", "ja3s": "8d2a028aa94425f76ced7826b1f39039" } - Let’s examine some of the key fields to better understand how we can use them to analyze SSL/TLS connections on our own network. For a full listing, check out the official Zeek documentation.
- uid (e.g., CY188jv7sKlzAMNNc): This is equivalent to the uid or unique ID that’s used to correlate activity across conn.log and other Zeek logs.
- version (e.g., TLSv12): The SSL/TLS version the server chose for the connection.
- cipher (e.g., TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256): The SSL/TLS cipher suite the server chose for the connection.
- curve (e.g., secp256r1): The elliptic curve the server chose when using ECDH/ECDHE.
- server_name (e.g., twitter.com): The hostname listed in the Server Name Indication (SNI) extension.
- resumed (e.g., false): Indicates if the connection was resumed or not.
- next_protocol (e.g., h2): The protocol the server chose to perform over the secure connection via application-layer protocol negotiation (ALPN). In this example, “h2” refers to “HTTP/2.”
- established (e.g., true): Indicates whether the encrypted session was successfully established.
- cert_chain_fuids (e.g., FzHo3q14WMvHTjCc79, F5woTi2O9Ea05rRxhj): The unique file IDs (fuids) of all certificates offered by the server.
- subject (e.g., CN=twitter.com,OU=atla,O=Twitter\\, Inc.,L=San Francisco,ST=California,C=US): The subject of the certificate.
- issuer (e.g., CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US): The issuer/signer of the certificate.
- validation_status (e.g., ok): Indicates the status of a certificate (e.g., ok, expired, self-signed, etc.).
- We can find the associated X.509 certificates by searching on the cert_chain_fuids. Below is a sample x509.log file in JSON format for each of the two cert_chain_fuids from our ssl.log above.
cert_chain_fuid: FzHo3q14WMvHTjCc79{ "ts": 1600552002.062889, "id": "FzHo3q14WMvHTjCc79", "certificate.version": 3, "certificate.serial": "0FBABC6DFD509287B4B260CE67C6F292", "certificate.subject": "CN=twitter.com,OU=atla,O=Twitter\\, Inc.,L=San Francisco,ST=California,C=US", "certificate.issuer": "CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US", "certificate.not_valid_before": 1580968800, "certificate.not_valid_after": 1612548000, "certificate.key_alg": "rsaEncryption", "certificate.sig_alg": "sha256WithRSAEncryption", "certificate.key_type": "rsa", "certificate.key_length": 2048, "certificate.exponent": "65537", "san.dns": [ "twitter.com", "www.twitter.com" ], "basic_constraints.ca": false }cert_chain_fuid: F5woTi2O9Ea05rRxhj
{ "ts": 1600552002.062889, "id": "F5woTi2O9Ea05rRxhj", "certificate.version": 3, "certificate.serial": "04E1E7A4DC5CF2F36DC02B42B85D159F", "certificate.subject": "CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US", "certificate.issuer": "CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US", "certificate.not_valid_before": 1382461200, "certificate.not_valid_after": 1855846800, "certificate.key_alg": "rsaEncryption", "certificate.sig_alg": "sha256WithRSAEncryption", "certificate.key_type": "rsa", "certificate.key_length": 2048, "certificate.exponent": "65537", "basic_constraints.ca": true, "basic_constraints.path_len": 0 }Note: The first X.509 is the host server’s certificate and the second X.509 is the intermediate certificate that signed and issued the first. If we had skipped the steps in “Enable X.509 Logging of The Full Certificate Chain,” we would only have the X.509 certificate for the host server and not the intermediate certificate that issued it.
- Let’s examine some of the key fields to better understand how we can use them to analyze X.509 certificates on our own network. For a full listing, check out the official Zeek documentation.
- id (e.g., FzHo3q14WMvHTjCc79 / F5woTi2O9Ea05rRxhj): This is the cert_chain_fuid or unique file ID that’s used to correlate files across Zeek logs.
- certificate.version (e.g., 3): The certificate’s version number.
- certificate.serial (e.g., 0FBABC6DFD509287B4B260CE67C6F292 / 04E1E7A4DC5CF2F36DC02B42B85D159F): The certificate’s serial number.
- certificate.subject (e.g., CN=twitter.com,OU=atla,O=Twitter\\, Inc.,L=San Francisco,ST=California,C=US / CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US): The certificate’s subject.
- certificate.issuer (e.g., CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US / CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US): The issuer of the certificate. Note again that the first certificate was issued by the second certificate which itself was issued by a root certificate that is presumably trusted by the client.
- certificate.not_valid_before (e.g., 1580968800 / 1382461200): The timestamp in Unix epoch time for when the certificate is first valid.
- certificate.not_valid_after (e.g., 1612548000 / 1855846800): The timestamp in Unix epoch time for when the certificate is no longer valid.
- certificate.key_alg (e.g., rsaEncryption): The certificate’s encryption key algorithm.
- certificate.sig_alg (e.g., sha256WithRSAEncryption): The certificate’s signature algorithm.
- certificate.key_type (e.g., rsa): The certificate’s encryption key type.
- certificate.key_length (e.g., 2048): The certificate’s encryption key length.
- certificate.exponent (e.g., 65537): The certificate’s exponent, if it is an RSA-certificate.
- san.dns (e.g., twitter.com, www.twitter.com): The certificate’s list of Subject Alternative Name (SAN) DNS entries.
- basic_constraints.ca (e.g., false / true): This identifies whether or not the certificate is a certificate authority. Note again, that the first certificate is not while the second is.
- basic_constraints.path_len (e.g., 0): The certificate’s maximum path length.
We’ll discuss the remaining fields in the next section.
Fingerprint SSL/TLS with JA3
In Part VI, we learned how to leverage Zeek’s incredible File Analysis Framework to automatically hash and uniquely fingerprint all files on the network. This enabled us to easily identify known malicious files and quickly analyze unknown files by their hash. Since the goal of encrypted traffic is confidentiality, wouldn’t it be great to have a way to fingerprint encrypted SSL/TLS traffic to identify and investigate for malicious activity? Fortunately, the team at Salesforce took this idea and developed a fingerprinting method to uniquely identify encrypted TLS traffic using the information Zeek already captures in ssl.log. At a high level, by combining cleartext elements of the SSL/TLS negotiation (e.g. version, cipher, elliptic curve, etc.) and MD5 hashing them, they’ve created the JA3 fingerprint for the client side and the JA3S fingerprint for the server’s response.
Our sample ssl.log from above includes these ja3/ja3s fields.
- ja3 (b32309a26951912be7dba376398abc3b): Client-side JA3 fingerprint.
- ja3s (e.g., 8d2a028aa94425f76ced7826b1f39039): Server-side JA3S fingerprint.
So how do we integrate JA3 into Zeek? If you followed the “Install Additional Useful Packages” section in Part II: Zeek Package Manager then you’re all set! Your ssl.log will already have the JA3/JA3S fingerprints included. From there, you can use the JA3 fingerprints just as you would with the file hashes — searching your logs for known malicious JA3 fingerprints or searching on identified JA3 fingerprints to investigate notable activity. While there are JA3 evasion techniques, it remains a great method for analyzing encrypted SSL/TLS traffic and is increasingly supported by a number of security platforms.
Analyze Decrypted HTTP/2 traffic
What if you’re one of the few organizations that actively decrypts web traffic through a network appliance such as a firewall? If you’re already doing this, you can also send a copy of that decrypted traffic to Zeek to have it inspected as usual. You’ll notice additional logging in http.log, but you may find that you’re not seeing everything you expected — especially from the internet’s major destinations. It turns out that is due to most major web properties using HTTP/2 instead of the traditional HTTP/1.1. Zeek cannot natively parse HTTP/2, but fortunately for us, the team at MITRE has developed an excellent Zeek package for analyzing HTTP/2 traffic.
The following steps detail how to install the HTTP/2 analyzer package. As a reminder, for this to be useful, you will need to already be decrypting HTTPS traffic outside of Zeek and then sending a copy of this decrypted traffic to Zeek. Zeek cannot decrypt traffic on its own.
- As root/sudo, install the required dependencies.
sudo apt-get install libbrotli-dev libnghttp2-dev - As the zeek user, stop zeek.
zeekctl stop - As the zeek user, use zkg to install the HTTP/2 analyzer package.
zkg install zeek/mitrecnd/bro-http2 The following packages will be INSTALLED: zeek/mitrecnd/bro-http2 (0.5.1) Verify the following REQUIRED external dependencies: (Ensure their installation on all relevant systems before proceeding): from zeek/mitrecnd/bro-http2 (0.5.1): libnghttp2>=1.11.0 libbrotlidec>=1.0.0 Proceed? [Y/n] y Running unit tests for "zeek/mitrecnd/bro-http2" Installing "zeek/mitrecnd/bro-http2"......................... Installed "zeek/mitrecnd/bro-http2" (0.5.1) Loaded "zeek/mitrecnd/bro-http2" - Edit/opt/zeek/share/zeek/site/local.zeek and add the following lines. Note that you must specifically add these lines to local.zeek as the package is not enabled by simply using @load packages (which typically enables all Zeek packages):
@load http2 @load http2/intel - As the zeek user, run zeekctl deploy to apply configurations and run Zeek.
zeekctl deploy - Assuming you’re sending decrypted HTTP/2 traffic to Zeek, you should now see a new http2.log in /opt/zeek/logs/current. Below is a sample http2.log file in JSON format. Note the version field’s value of “2.0” confirming that this is in fact HTTP/2 traffic.
{ "ts": 1600987697.997372, "uid": "C6IFL331Q4qxw5sacf", "id.orig_h": "10.0.1.9", "id.orig_p": 55837, "id.resp_h": "172.217.9.164", "id.resp_p": 443, "stream_id": 1, "method": "GET", "host": "www.google.com", "uri": "/", "version": "2.0", "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36", "request_body_len": 0, "response_body_len": 220105, "status_code": 200, "status_msg": "", "encoding": "br", "push": false }Note: Even after following the steps above, you may find that you don’t see much Google traffic. This is because Google tends to use the QUIC protocol and while there is a Zeek package for analyzing QUIC traffic, it only logs metadata about the activity as it cannot perform decryption directly. Unless you have a way to directly decrypt QUIC, a workaround is to block QUIC with a firewall and force Google to use TLS and HTTP/2 instead. From there, you can decrypt web traffic as normal and send this decrypted traffic directly to Zeek for analysis.
Understand ssh.log
Just as with SSL/TLS, while Zeek cannot directly decrypt SSH traffic, it can still observe and capture details from the SSH handshake — even going as far as heuristically determining whether an authentication attempt was successful.
- Take a look at your own ssh.log and note the encrypted SSH connections and details that are captured. Below is a sample ssh.log file in JSON format.
{ "ts": 1600955934.041752, "uid": "C6JIXO2J1rHy9J0jAd", "id.orig_h": "10.0.1.99", "id.orig_p": 60774, "id.resp_h": "10.0.1.102", "id.resp_p": 22, "version": 2, "auth_success": true, "auth_attempts": 2, "client": "SSH-2.0-OpenSSH_8.1", "server": "SSH-2.0-OpenSSH_7.4", "cipher_alg": "chacha20-poly1305@openssh.com", "mac_alg": "umac-64-etm@openssh.com", "compression_alg": "none", "kex_alg": "curve25519-sha256", "host_key_alg": "ecdsa-sha2-nistp256", "host_key": "15:44:af:61:9f:fd:d6:51:f4:e8:35:3e:90:e7:9d:d5", "hasshVersion": "1.1", "hassh": "ec7378c1a92f5a8dde7e8b7a1ddf33d1", "hasshServer": "6832f1ce43d4397c2c0a3e2f8c94334e", "cshka": "ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa", "hasshAlgorithms": "curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c;chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com;umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1;none,zlib@openssh.com,zlib", "sshka": "ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519", "hasshServerAlgorithms": "curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1;chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc;umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1;none,zlib@openssh.com" } - Let’s examine some of the key fields to better understand how we can use them to analyze SSH connections on our own network. For a full listing, check out the official Zeek documentation.
- uid (e.g., CY188jv7sKlzAMNNc): This is equivalent to the uid or unique ID that’s used to correlate activity across conn.log and other Zeek logs.
- version (e.g., 2): The SSH version in use.
- auth_success (e.g., true): Using heuristics, Zeek determines whether the SSH session authenticated successfully.
- auth_attempts (e.g., 2): Using heuristics, Zeek determines the number of authentication attempts made.
- client (e.g., SSH-2.0-OpenSSH_8.1): The SSH client in use.
- server (e.g., SSH-2.0-OpenSSH_7.4): The SSH server in use.
- cipher_alg (e.g., chacha20-poly1305@openssh.com): The encryption algorithm used.
- mac_alg (e.g., umac-64-etm@openssh.com): The signing algorithm (MAC) used.
- compression_alg (e.g., none): The compression algorithm used.
- kex_alg (e.g., curve25519-sha256): The key exchange algorithm used.
- host_key_alg (e.g., ecdsa-sha2-nistp256): The server’s key algorithm.
- host_key (e.g., 15:44:af:61:9f:fd:d6:51:f4:e8:35:3e:90:e7:9d:d5): The server’s key fingerprint.
We’ll discuss the remaining fields in the section below.
- Zeek also includes built-in scripts that take advantage of its SSH capabilities — detecting SSH to/from specific countries and detecting brute force attacks. You can confirm these are enabled by viewing /opt/zeek/share/zeek/site/local.zeek and verifying the @load statements are uncommented as they are below.
# If you have GeoIP support built in, do some geographic detections and # logging for SSH traffic. @load protocols/ssh/geo-data # Detect hosts doing SSH bruteforce attacks. @load protocols/ssh/detect-bruteforcing
Fingerprint SSH with HASSH
Similar to how JA3 can be used to fingerprint SSL/TLS connections, HASSH can be used to fingerprint SSH connections. Just as with JA3, you can use these fingerprints to profile and identify suspicious SSH activity. Our sample ssh.log from above includes HASSH fields that we’ll review below.
- hasshVersion (e.g., 1.1): The HASSH version in use.
- hassh (e.g., ec7378c1a92f5a8dde7e8b7a1ddf33d1): The client-side HASSH fingerprint.
- hasshServer (e.g., 6832f1ce43d4397c2c0a3e2f8c94334e): The server-side HASSH fingerprint.
- cshka (e.g., ecdsa-sha2-nistp256-cert-v01@openssh.com): The client host key algorithms.
- hasshAlgorithms (e.g., curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c;chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com;umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1;none,zlib@openssh.com,zlib): The concatenated list of client-side algorithms including key exchange methods, encryption, message auth, and compression. The client-side HASSH fingerprint is a result of taking the MD5 hash of this list.
- sshka (e.g., ssh-rsa, rsa-sha2-512): The server host key algorithms.
- hasshServerAlgorithms (e.g., curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1;chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc;umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1;none,zlib@openssh.com): The concatenated list of server-side algorithms including key exchange methods, encryption, message auth, and compression. The server-side HASSH fingerprint is a result of taking the MD5 hash of this list.
Up Next
In Part VIII of the series, we will send Zeek logs to Elasticsearch and create Kibana queries.
