ClusterBackend = Broker

If you manage a non-Zeekctl Zeek setup, load policy/frameworks/cluster/backend/zeromq or policy/frameworks/cluster/backend/broker. If you forget to do so, Zeek will recognize that Cluster::node is set (e.g. the CLUSTER_NODE environment variable or Cluster::node=... is set) but no cluster backend was chosen and will terminate with a fatal error:

fatal error in <...>/main.zeek, line 425: Cluster::node set to 'worker-1', but Cluster::backend is Cluster::CLUSTER_BACKEND_NONE - please select a cluster backend to use.

If you used Zeekctl on a single physical system or virtual machine previously, this should all be transparent to you.

New Functionality

• After recently-merged fixes to the node formula for Homebrew, JavaScript support via ZeekJS can now be built on macOS using that installation of Node.

• A new TapAnalyzer class was added allowing to tap into all packets delivered to child analyzers attached to session adapters.

• Two new hooks, Cluster::on_subscribe() and Cluster::on_unsubscribe() have been added to allow observing Subscribe() and Unsubscribe() calls on backends by Zeek scripts.

• The ability to control the length of strings and containers in log output was added. The maximum length of individual log fields can be set, as well as the total length of all string or container fields in a single log record. This feature is controlled via four new script-level variables:

  Log::default_max_field_string_bytes Log::default_max_total_string_bytes Log::default_max_field_container_elements Log::default_max_total_container_elements

  When one of the field limits is reached, the individual field is truncated. When one of the total limits is reached, all further strings will returned as empty and all further container elements will not be output. See the documentation for those variables for more detail.

  The above variables control the truncation globally, but they can also be set for log streams individually. This is controlled by variables with the same names that can be set when the log stream is created.

  Two new weirds were added to report the truncation: log_string_field_truncated and log_container_field_truncated. New metrics were added to track how many truncations have occurred: zeek_log_writer_truncated_string_fields_total and zeek_log_writer_truncated_containers_total. The metrics are reported for each log stream.

• There exists a new configuration file <PREFIX>/etc/zeek/zeek.conf and a small executable called zeek-systemd-generator that acts as a systemd unit file generator. These two pieces allow for quickly running an opinionated single node cluster deployment using systemd. See the tools/systemd-generator/README.md and tools/systemd-generator/etc/zeek/zeek.conf files for more information.

  If you've previously built Zeek process management by hand or with tools like supervisord, runit, s6, etc on Linux, this might be a viable option to explore.

  This is all Linux specific right now. However, the zeek.conf file is OS-agnostic to keep options open for re-use it elsewhere.

• The DNS analyzer now returns the set of parameters for SVCB data. It previously handled SVCB packets, but omitted the parameters while parsing.

• The QUIC analyzer now raises a QUIC::discarded_packet event when a packet with fixed_bit set to 0 is encountered. Such an occurrence is included in the QUIC history as X. This functionality can be controlled with QUIC::max_discarded_packet_events. Setting this variable to -1 disables the QUIC::discarded_packet event.

• Added SHA224, SHA384, and SHA512 calculation BiFs: sha224_hash, sha224_hash_init, sha224_hash_update, sha224_hash_finish. sha384_hash, sha384_hash_init, sha384_hash_update, sha384_hash_finish. sha512_hash, sha512_hash_init, sha512_hash_update, and sha512_hash_finish.

• Added SHA224, SHA384 and SHA512 file hash analyzers. The analyzers are not enabled by default.

• Added the TCP::raw_options BiF, which returns the list of TCP options in the current packet, including their raw data.

• For SNMPv3, Zeek now parses the Security Parameters for the User-Based Security model. If present, the data is added to the new user_security_parameters field of the SNMP::HeaderV3 record.

• The QUIC analyzer now raises QUIC::short_header_packet_threshold_crossed() for every binary logarithmic threshold for QUIC packets with short headers. Each threshold is added to the history as O or o. E.g., a history containing oOoO indicates that originator and responder each send at least 3 short packets, while oOOoooooo would indicate the originator sent at least 3 short packets early in the connection, while the responder sent at least 127 packets.

• Added SHA256 calculation BiFs: sha512_hash, sha512_hash_init, sha512_hash_update, and sha512_hash_finish.

• Added VLAN PCP and DEI bits when parsing packets. This is exposed alongside vlan and inner_vlan in a packet's l2_hdr when scripting, if a VLAN header was present.

• The DNS analyzer now supports parsing Dynamic Update messages (RFC 2136) and reporting information about them into dns.log. The answer field in the log will contain the prerequisites and updates for the dynamic update requests, whereas the query field logs the zone name for the update request.

  This also adds new opcode and opcode_name fields to the DNS log. Previously Zeek was only handling opcode 0 (normal requests) so that information wasn't pertinent, but with the addition of dynamic updates we're now handling a second opcode. The new policy script policy/protocols/dns/disable-opcode-log-fields.zeek can be loaded to remove these fields from the log if they are not desired.

• The policy/misc/dump-events.zeek script now features a DumpEvents::use_json boolean toggle, false by default, to report Zeek's events in JSON format.

• The list of HTTP methods accepted by dynamic protocol detection was updated to add some missing methods: BIND, LINK, MKCALENDAR, MKREDIRECTREF, REBIND, UNBIND, UNLINK.

• The Docker images generated by Zeek's CI process now contain OCI-approved annotation labels containing various metadata about the image. See https://github.com/opencontainers/image-spec/blob/main/annotations.md for more details about the labels.

Changed Functionality

• A Zeekctl-managed Zeek cluster now uses the ZeroMQ cluster backend by default.

• The manager node of a Zeekctl-managed Zeek cluster now listens on 127.0.0.1:27759 for WebSocket connections by default. This can be disabled with the UseWebSocket setting in zeekctl.cfg, or adapted using the WebSocketHost and WebSocketPort settings.

• The var-extraction-uri.zeek policy does not include the path in the uri_vars field anymore.

• The get_current_packet_header() BIF now populates the returned record also for fragmented IP datagrams.

• The QUIC parser discards packets with the fixed_bit field set to 0, rather than continuing to parse and potentially running into analyzer violations.

• The Notice framework's suppression implementation was changed to batch suppression information for up up to 10msec before distributing it to other nodes in a Zeek cluster. This reduces cluster communication overhead when there's a storm of notices generated. Two new options, Notice::suppression_batch_period and Notice::suppression_batch_max_size can be used to tune this behavior. See their documentation for details. This most notably reduces CPU usage and cluster event traffic during cluster startup.

• Non-printable ASCII control characters now always result in \uXXXX characters when logging them as JSON. This follows the JSON standard more closely. Previously, they would have been logged in the \xXX format that Zeek typically uses. For example, an HTTP request with a non-printable character in the URI would have a uri field that changes from:

  "uri":"/non_printable_\x07"

  to

  "uri":"/non_printable_\u0007"

• For SNMPv3, the community field of snmp.log now contains the SNMPv3 username (if present). Previously, the field was empty for all SNMPv3 connections.

• The x509.log deduplication logic has been reimplemented with explicit calls to Cluster::publish(), rather than relying &backend tables using Broker.

• Sanitizer flags passed to a Zeek build with configure's --sanitizers argument are now automatically passed to plugins built against that Zeek source tree.

• The hash-all-files policy script now enables SHA256 logging, in addition to MD5 and SHA1.

• The LDAP analyzer was fixed to correctly escape non-ASCII characters when logging filter strings.

• The AF_Packet packet source now reports a warning if you attempt to use it on a loopback interface. This is because AF_PACKET will receive duplicates of every packet when run on a loopback interface and this can be confusing.

• The GSSAPI analyzer gained some fixes to reliably process Kerberos data when that data isn't preceded by an OID. This is valid a ASN.1 structure that wasn't being handled correctly.

• SSL certificates are now correctly logged even when protocol violations occur, assuming the certificate was encountered prior to the violation.

Deprecated Functionality

• The &backend and &broker_store attributes for script-layer tables have been deprecated. These are only functional with the Broker cluster backend and their semantics and guarantees considered too loose. There is ongoing discussion and experiments how to provide state propagation and synchronization across cluster processes. Explicit topic-based state propagation using Cluster::publish(), or switching to the storage framework for persistence are available today.

  If you've used &backend and &broker_store in your scripts and find it hard or inconvenient to replace them, please reach out with your use-cases, on GitHub or Slack. We're happy to support.

  Some references to GitHub discussion and prototypical ideas:

  https://github.com/zeek/zeek/discussions/5020 https://github.com/zeek/zeek/pull/5040

• In Zeek 9.1, returning a value from a hook will become an error. Return values from a hooks were previously silently discarded. This behavior was found surprising in relation with the break statement available in hooks.

• The zeek::util::json_escape_utf8 methods are being replaced by a new escape_utf8 method in order to fix a bug with how the escape_printable_controls argument was implemented. Previously, it was implemented in the reverse of what was intuitive given the name. It would escape the control characters when false, and not escape them when true. The new method has the correct semantics, as well as a new argument for controlling whether non-printable control characters are escaped.

• The zeromq/connect script has been deprecated. Load the frameworks/cluster/backend/zeromq package directly instead.

• The rfb_auth_result event from the RFB analyzer was deprecated and replaced with a new rfb_authentication_result event. This new event sets the correct value in the rfb$auth field in the connection.