We'd like to thank Andrew Raffman (@andyraf), @arm7mm, Artyom Kalabukhov (@predator89090), Benjamin Grap (@blightzero), @biswajitutil, @cccs-jsjm, @chrisjlly, Craig P (@multicast-bytes), @DigiAngel, Mike Dopheide (@dopheide-esnet), Edoardo Mich (@edoardomich), @franky-m, Fupeng Zhao (@AmazingPP), Jan Grashöfer (@J-Gras), Klemens Nanni (@klemens-ya), Klemens Nanni (@klemensn), @kshitiz56, Mark Overholser (@markoverholser), Mohan Dhawan (@Mohan-Dhawan), Peter Cullen (@pbcullen), RyugaXhypeR (@RyugaXhypeR), Seth Hall (@sethhall), Yacin Nadji (@ynadji), Zhang Liang (@JosenLiang), Zach Robinette (@zrobinette12) and @zzzdong for their contributions to this release.

Breaking Changes

• A standalone zeek invocation on the command line will now default to running a No-op Cluster backend (Cluster::CLUSTER_BACKEND_NONE). Previously this defaulted to Broker without having any real effect. For a multi-process / cluster deployment of Zeek, a cluster backend policy script needs to be loaded explicitly.

  Zeekctl has been updated to use the ZeroMQ cluster backend by default and should require no other action. If you encounter issues with ZeroMQ, it is possible to re-enable the Broker cluster backend by adding the following entry into zeekctl.cfg and running `zeekctl deploy``:

  ClusterBackend = Broker

  If you manage a non-Zeekctl Zeek setup, load policy/frameworks/cluster/backend/zeromq or policy/frameworks/cluster/backend/broker. If you forget to do so, Zeek will recognize that Cluster::node is set (e.g. the CLUSTER_NODE environment variable or Cluster::node=... is set) but no cluster backend was chosen and will terminate with a fatal error:

  fatal error in <...>/main.zeek, line 425: Cluster::node set to 'worker-1', but Cluster::backend is Cluster::CLUSTER_BACKEND_NONE - please select a cluster backend to use.

  If you used Zeekctl on a single physical system or virtual machine previously, this should all be transparent to you.

• The Broker-specific &backend and &broker_store attributes will result in startup errors when a non-Broker cluster backend is selected. See the entry about their deprecation below for more details.

• Python 3.10 is now required for Zeek and all of its associated subprojects.

• The &optional script attribute will now error when applied to anything that's not a record field. Previously, this would have surprising behavior.

• The BinPAC, Bifcl, and Gen-ZAM tools have all moved directly into the Zeek repo, which should ease maintenance on them a bit. They were moved from the auxil directory to the tools directory. Along with this, the --gen-zam argument for configure was removed and the internal version will always be used.

• The zeek-af_packet-plugin git submodule was moved directly into the Zeek repo. This used to live in the auxil directory, after having moved there from an external plugin. It is now automatically built as part of the main Zeek build whenever building on Linux.

• The global g_frame_stack C++ variable was removed. This variable was used to track the current stack of script-function frames, mostly for the built-in script debugger. We realized that we were also tracking this information in the global call_stack variable, used by the script profiler. These two have now been combined together into call_stack. Plugins using g_frame_stack will need to be adapted to use the other variable. We opted to not use a deprecation cycle for this due to the complication with managing the state across both variables.

• The unique_id and unique_id_from BIFs now return 96-bit UIDs, which makes them consistent with other UIDs created by Zeek such as connection IDs. They were previously returning shorter 64-bit UIDs.

• The metrics zeek_msgthread_pending_messages_in_buckets and zeek_msgthread_pending_messages_out_buckets had labels that did not conform with Prometheus' checks via promtool. The labels have been renamed from le to leq, given that they technically are less-than-or-equal values.

• The iosource/Packet.h header no longer depends on libpcap being installed and no longer directly includes the pcap.h header. This may cause some knock-on effects in building plugins that were depending on this header and others that it included indirectly.

• External plugins do not implicitly require OpenSSL and Broker anymore. Starting with Zeek 8.1, plugins that depend on these libraries are now required to explicitly call find_package(OpenSSL REQUIRED) and pass OPENSSL_LIBRARIES as dependencies to the zeek_add_plugin() call:

  find_package(OpenSSL REQUIRED)

  zeek_add_plugin( ... INCLUDE_DIRS ${OPENSSL_INCLUDE_DIR} DEPENDENCIES ${OPENSSL_LIBRARIES} )

• We are revisiting removing support for MD5 hashing across the Zeek code base. In support of this, the following changes were made:

  • The host_key field in ssh.log was removed and replaced with a new host_key_fingerprint field. The old field held an MD5 hash of the host key. The new field holds a base64-encoded version of the hash of the host key, in the same format that the ssh-keygen -l command would return. It currently uses SHA256 for the hashing, so the output is of the format SHA256:<fingerprint>. A new policy/protocols/ssh/md5-host-key-logging.zeek script was added to restore the old field if desired.

  • The auth_ticket and new_ticket fields in krb.log were removed and replaced with new auth_ticket_sha256 and new_ticket_sha256 fields. The old fields held MD5 hashes of the tickets and the new fields hold SHA256 hashes of the same. A new policy/protocols/krb/md5-ticket-logging.zeek script was added to restore the old fields if desired.

  • A new mime_content_hash_sha256 event was added to send a SHA256 version of the content in MIME messages. This is similar to the existing mime_content_hash event that sends MD5 versions.

  • The MD5 IP anonymizers (RANDOM_MD5 and PREFIX_PRESERVING_MD5) have been deprecated in favor of new SHA256 anonymizers.

  • The md5_hmac BIF has been deprecated in favor of a new sha256_hmac BIF.

New Functionality

• After recently-merged fixes to the node formula for Homebrew, JavaScript support via ZeekJS can now be built on macOS using that installation of Node.

• A new TapAnalyzer class was added allowing to tap into all packets delivered to child analyzers attached to session adapters.

• Two new hooks, Cluster::on_subscribe() and Cluster::on_unsubscribe() have been added to allow observing Subscribe() and Unsubscribe() calls on backends by Zeek scripts.

• The ability to control the length of strings and containers in log output was added. The maximum length of individual log fields can be set, as well as the total length of all string or container fields in a single log record. This feature is controlled via four new script-level variables:

  Log::default_max_field_string_bytes Log::default_max_total_string_bytes Log::default_max_field_container_elements Log::default_max_total_container_elements

  When one of the field limits is reached, the individual field is truncated. When one of the total limits is reached, all further strings will returned as empty and all further container elements will not be output. See the documentation for those variables for more detail.

  The above variables control the truncation globally, but they can also be set for log streams individually. This is controlled by variables with the same names that can be set when the log stream is created.

  Two new weirds were added to report the truncation: log_string_field_truncated and log_container_field_truncated. New metrics were added to track how many truncations have occurred: zeek_log_writer_truncated_string_fields_total and zeek_log_writer_truncated_containers_total. The metrics are reported for each log stream.

• There exists a new configuration file <PREFIX>/etc/zeek/zeek.conf and a small executable called zeek-systemd-generator that acts as a systemd unit file generator. These two pieces allow for quickly running an opinionated single node cluster deployment using systemd. See the tools/systemd-generator/README.md and tools/systemd-generator/etc/zeek/zeek.conf files for more information.

  If you've previously built Zeek process management by hand or with tools like supervisord, runit, s6, etc on Linux, this might be a viable option to explore.

  This is all Linux specific right now. However, the zeek.conf file is OS-agnostic to keep options open for re-use it elsewhere.

• The DNS analyzer now returns the set of parameters for SVCB data. It previously handled SVCB packets, but omitted the parameters while parsing.

• The QUIC analyzer now raises a QUIC::discarded_packet event when a packet with fixed_bit set to 0 is encountered. Such an occurrence is included in the QUIC history as X. This functionality can be controlled with QUIC::max_discarded_packet_events. Setting this variable to -1 disables the QUIC::discarded_packet event.

• Added SHA224, SHA384, and SHA512 calculation BiFs: sha224_hash, sha224_hash_init, sha224_hash_update, sha224_hash_finish. sha384_hash, sha384_hash_init, sha384_hash_update, sha384_hash_finish. sha512_hash, sha512_hash_init, sha512_hash_update, and sha512_hash_finish.

• Added SHA224, SHA384 and SHA512 file hash analyzers. The analyzers are not enabled by default.

• Added the TCP::raw_options BiF, which returns the list of TCP options in the current packet, including their raw data.

• For SNMPv3, Zeek now parses the Security Parameters for the User-Based Security model. If present, the data is added to the new user_security_parameters field of the SNMP::HeaderV3 record.

• The QUIC analyzer now raises QUIC::short_header_packet_threshold_crossed() for every binary logarithmic threshold for QUIC packets with short headers. Each threshold is added to the history as O or o. E.g., a history containing oOoO indicates that originator and responder each send at least 3 short packets, while oOOoooooo would indicate the originator sent at least 3 short packets early in the connection, while the responder sent at least 127 packets.

• Added SHA256 calculation BiFs: sha512_hash, sha512_hash_init, sha512_hash_update, and sha512_hash_finish.

• Added VLAN PCP and DEI bits when parsing packets. This is exposed alongside vlan and inner_vlan in a packet's l2_hdr when scripting, if a VLAN header was present.

• The DNS analyzer now supports parsing Dynamic Update messages (RFC 2136) and reporting information about them into dns.log. The answer field in the log will contain the prerequisites and updates for the dynamic update requests, whereas the query field logs the zone name for the update request.

  This also adds new opcode and opcode_name fields to the DNS log. Previously Zeek was only handling opcode 0 (normal requests) so that information wasn't pertinent, but with the addition of dynamic updates we're now handling a second opcode. The new policy script policy/protocols/dns/disable-opcode-log-fields.zeek can be loaded to remove these fields from the log if they are not desired.

• The policy/misc/dump-events.zeek script now features a DumpEvents::use_json boolean toggle, false by default, to report Zeek's events in JSON format.

• The list of HTTP methods accepted by dynamic protocol detection was updated to add some missing methods: BIND, LINK, MKCALENDAR, MKREDIRECTREF, REBIND, UNBIND, UNLINK.

• The Docker images generated by Zeek's CI process now contain OCI-approved annotation labels containing various metadata about the image. See https://github.com/opencontainers/image-spec/blob/main/annotations.md for more details about the labels.

Changed Functionality

• A Zeekctl-managed Zeek cluster now uses the ZeroMQ cluster backend by default.

• The manager node of a Zeekctl-managed Zeek cluster now listens on 127.0.0.1:27759 for WebSocket connections by default. This can be disabled with the UseWebSocket setting in zeekctl.cfg, or adapted using the WebSocketHost and WebSocketPort settings.

• The var-extraction-uri.zeek policy does not include the path in the uri_vars field anymore.

• The get_current_packet_header() BIF now populates the returned record also for fragmented IP datagrams.

• The QUIC parser discards packets with the fixed_bit field set to 0, rather than continuing to parse and potentially running into analyzer violations.

• The Notice framework's suppression implementation was changed to batch suppression information for up up to 10msec before distributing it to other nodes in a Zeek cluster. This reduces cluster communication overhead when there's a storm of notices generated. Two new options, Notice::suppression_batch_period and Notice::suppression_batch_max_size can be used to tune this behavior. See their documentation for details. This most notably reduces CPU usage and cluster event traffic during cluster startup.

• Non-printable ASCII control characters now always result in \uXXXX characters when logging them as JSON. This follows the JSON standard more closely. Previously, they would have been logged in the \xXX format that Zeek typically uses. For example, an HTTP request with a non-printable character in the URI would have a uri field that changes from:

  "uri":"/non_printable_\x07"

  to

  "uri":"/non_printable_\u0007"

• For SNMPv3, the community field of snmp.log now contains the SNMPv3 username (if present). Previously, the field was empty for all SNMPv3 connections.

• The x509.log deduplication logic has been reimplemented with explicit calls to Cluster::publish(), rather than relying &backend tables using Broker.

• Sanitizer flags passed to a Zeek build with configure's --sanitizers argument are now automatically passed to plugins built against that Zeek source tree.

• The hash-all-files policy script now enables SHA256 logging, in addition to MD5 and SHA1.

• The LDAP analyzer was fixed to correctly escape non-ASCII characters when logging filter strings.

• The AF_Packet packet source now reports a warning if you attempt to use it on a loopback interface. This is because AF_PACKET will receive duplicates of every packet when run on a loopback interface and this can be confusing.

• The GSSAPI analyzer gained some fixes to reliably process Kerberos data when that data isn't preceded by an OID. This is valid a ASN.1 structure that wasn't being handled correctly.

• SSL certificates are now correctly logged even when protocol violations occur, assuming the certificate was encountered prior to the violation.

Deprecated Functionality

• The &backend and &broker_store attributes for script-layer tables have been deprecated. These are only functional with the Broker cluster backend and their semantics and guarantees considered too loose. There is ongoing discussion and experiments how to provide state propagation and synchronization across cluster processes. Explicit topic-based state propagation using Cluster::publish(), or switching to the storage framework for persistence are available today.

  If you've used &backend and &broker_store in your scripts and find it hard or inconvenient to replace them, please reach out with your use-cases, on GitHub or Slack. We're happy to support.

  Some references to GitHub discussion and prototypical ideas:

  https://github.com/zeek/zeek/discussions/5020 https://github.com/zeek/zeek/pull/5040

• In Zeek 9.1, returning a value from a hook will become an error. Return values from a hooks were previously silently discarded. This behavior was found surprising in relation with the break statement available in hooks.

• The zeek::util::json_escape_utf8 methods are being replaced by a new escape_utf8 method in order to fix a bug with how the escape_printable_controls argument was implemented. Previously, it was implemented in the reverse of what was intuitive given the name. It would escape the control characters when false, and not escape them when true. The new method has the correct semantics, as well as a new argument for controlling whether non-printable control characters are escaped.

• The zeromq/connect script has been deprecated. Load the frameworks/cluster/backend/zeromq package directly instead.

• The rfb_auth_result event from the RFB analyzer was deprecated and replaced with a new rfb_authentication_result event. This new event sets the correct value in the rfb$auth field in the connection.