lnav v0.14.0

Features:

• The Filter configuration panel in the TUI now supports editing the minimum log level and minimum/maximum times for the current view. Previously, only the :set-min-log-level, :hide-lines-before, and :hide-lines-after commands could be used to set the values. Pressing l will create/set the log level. Pressing m will create/set the minimum time and pressing Shift + M will create/set the maximum time. Live preview has also been added to show which lines will be filtered out when the min/max is applied.

• The -S/--since and -U/--until flags have been added to limit how much of a log file is indexed. Supported values are relative (e.g. "yesterday", "30 min ago") or absolute local times ("2020-01-01"). The values on the main command-line set the defaults for all values that are opened. The :open command supports the same options to change the values from the default for a particular file. Files with content that lie completely outside of the cutoff will be closed to reduce resource usage.

• Broadened support for viewing files on remote hosts. The "tailer" program that is transferred to the remote host to monitor files and perform other tasks has been translated to Python3. If the remote host does not have Python, lnav will fall back to the APE binary.

• The "Files" panel now shows a progress bar for each file as it is being indexed and finishes with a check-mark if indexing was successful, a warning sign if the file has some notes, or an error mark if something else happened.

• Introducing "Log-Oriented Debugging", a collection of features to streamline mapping log messages back to the source code that generated them. For example, given the log message "Hello, Steve!" and the source directory containing the log statement. lnav can find the line of code that generated the message, such as logging.info("Hello, %s!", name), and determine the value of the substituted variables (name => Steve). This functionality is implemented using the log2src project. The following features have been added in support of this functionality:

  • The :add-source-path command was added to specify the source directories to be scanned for log statements.
  • Log formats can now specify source file/line and thread ID with the src-file-field, src-line-field, and thread-id-field properties. These fields can then be accessed in the SQL vtables as log_src_file, log_src_line, and log_thread_id.
  • The :breakpoint, :toggle-breakpoint, and :clear-breakpoints commands have been added to support setting/clearing breakpoints for log messages. The CTRL-B shortcut toggles a breakpoint on the focused line in the LOG view. Also, if the log format specifies source file/line fields, the first character of the source file will be underlined and can be clicked to toggle a breakpoint. Once breakpoints have been added, you can press F7/F8 to move to the previous/next log message that matches a breakpoint.

• The all_opids and all_thread_ids virtual tables have been added to make it simple to discover all of the operations and threads across all log files. The all_opids table also supports setting a description for an operation using through an UPDATE.

• The :xopen command will now open text files in an external editor. To open the file at a particular line/column, add a URL fragment of the form L<line>C<column>.

• When opening the contents of the prompt in an external editor (CTRL+O), the cursor position will be preserved, if possible.

• The external-editor configuration has been expanded with extra properties to help lnav choose the right one to use:

  • The config-dir property specifies the name of the directory that stores the editor's configuration in a source tree. If the directory is found in an ancestor of the path to be opened, and it has the most recent modified time, the associated editor will be used.
  • The prefers property is a regular expression that will be tested against the full path to be opened. If matched, that editor will be chosen.

• The :external-access command has been added to open a localhost HTTP port that can be used to remotely control lnav. Requests can be sent to execute commands and poll for changes in the view state. When the external port is open, a globe icon (🌐) is displayed in the top-right corner. Clicking that icon will open a URL in a browser and log you into the server. The :external-access-login command can also be used to login.

• Custom "Apps" can be added to the "external access" server to provide custom browser-based user interfaces to lnav. See the "External Access" documentation online for more details.

• The ;.save SQL command has been added that can save tables you have created to a SQLite database file. The tables that lnav creates have been moved to a separate in-memory DB, so the main DB should only contain your own tables/views/etc...

• The json_object_count_of() SQL aggregate function has been added to make it easy to create a JSON object where the values are the number of times a value has been seen.

• The :write-json-cols-to command has been added to write JSON output in a column-oriented fashion.

• The encode() and decode() SQL functions now accept html as an algorithm.

• The opid-field can now be set to a JSON array/object for JSON-lines logs. For example, the spans array in a Rust tracing log message. The OPID for the message will be computed by hashing the contents of the array or object and the description will be the container itself.

• An OPID can now be constructed from multiple fields by leaving the opid-field blank and creating a single opid/description definition with a format. The content of the format fields will then be hashed to create the OPID. The builtin log formats have been updated to use this when appropriate. For example, access_log now uses c_ip and cs_user_agent as the OPID.

  [!NOTE] If you want a description, but don't want it used as the OPID. You can set the opid/source field to "from-whole-msg" and the OPID will be computed from the contents of the log message.

• The duration-field log format property has been added to specify the field that contains a duration in the log message. If a duration is available, it will be used to calculate time spans in the TIMELINE view.

• The timestamp-point-of-reference log format property has been added to specify the related of the timestamp to the operation that the message refers to, either: start or send. This is used in conjunction with the message duration to determine the time span.

• The OPID for log messages is now shown in the parser details overlay (revealed by pressing p) in the LOG view.

• Added rust_tracing_log format from @richard-hajek.

• Added macosuni_log format that understands the output of the macOS log stream --style=ndjson command.

• Added the idea_log format from @segevfiner.

• The strace_log format has been improved to handle more output formats and the syscalls will now show up in the TIMELINE view.

• The strace:// URL-handler has been added to make it easier to run strace on an existing process. A host must be given and the path should be the PID, such as strace://localhost/1234.

• Added the nestable flag to the log format and theme highlight configurations to control whether a highlight can be applied to text that is covered by another highlight.

• Search tables are now included in the output of the :export-session-to command.

Breaking changes:

• All of lnav's SQLite tables have been moved to a separate in-memory database that is attached as lnav_db. You may need to update some of your SQL statements to qualify table names with lnav_db.. This change was made so that the main DB only contains user data that can be easily backed up to a new DB file.
• Timestamp columns and results from lnav time functions now have microsecond precision instead of millisecond.
• The "module format" functionality has been removed. This functionality tried to match log messages wrapped in another format (usually syslog), but it never really worked well and was impeding progress in other areas. Also, there have been many features added since the beginning that can serve the same use cases.
• The sudo_log format has been removed since it was a module-only format. Instead, a sudo_log search-table was added to the syslog_log format.

Interface changes:

• Mouse mode is now enabled by default.
• The CTRL+f hotkey has been remapped to the :toggle-filtering command.
• Aborting the prompt now requires two successive presses of Esc (a message will pop up on the right that mentions this). Since Esc is also used to close the completion popup, it was too easy to cancel the prompt. Pressing CTRL+] will still close the prompt immediately.
• The TIMELINE view has a few updates:
  • The header has been redesigned to be one line that shows the time increments at the current scale. This approach should more clearly convey the spans of time shown in the main part of the view. The previous design tried to show the overall time and the current time frame. But, the multi-line header was hard to interpret and didn't make it clear how large the time increments were.
  • Log files and threads are now shown in the view in addition to operations.
• The HIST view now shows the year and inserts a spacer row in-between gaps in time. The spacer row shows bullet points on a log scale to represent the amount of time in the gap.
• The SPECTRO view now shows the year in timestamps and uses additional colors to show the value range.
• The breadcrumb bar in the LOG view now includes the current thread, if defined.
• If there are background tasks, like the processing done by :add-source-path, a panel with progress bars for each operation will be shown just above the bottom status bar.
• The first line of a multi-line log message will now be shown in the header of the LOG view if the message has scrolled off the screen. When scrolled to the beginning of the content, the top line will indicate if any log messages were filtered out before the first message. Left-clicking on the header will scroll the view to the displayed line.
• In the Filters configuration panel, you can now create/edit the SQL expression filter by pressing the e key.
• The { hotkey and :prev-section command will now move to the first line of a multi-line log message if the focused line is in the middle of the message.
• The } hotkey and :next-section command will now move to the next log message if the focused line is in the middle of a multi-line message.
• The parser-details overlay now mentions any search tables that match the focused line. Now, you don't have to remember the names of the tables.
• If a text file contains invalid UTF-8 content, the invalid bytes will be shown as the replacement character (�) and, when the line is focused, an overlay will show a hex dump of the line.

Bug Fixes:

• If a file path contains a hash (#), check if the path exists before splitting around the hash and treating it as an internal file location.
• The initialization sequence has been cleaned up to hopefully make it more consistent/reliable. This concerns the sequence of loading files and executing commands from the command-line.
• In the SPECTRO view, you can now move to the next/ previous row with bookmarks. Note that this view synchronizes bookmarks with the LOG view.
• Session state should no longer override commands passed on the command-line.

Internal:

• Added operation IDs (OpIDs) to lnav's internal logging to make it easier to identify the high-level operations that are being performed using the TIMELINE view. While this is mainly useful to the authors of lnav, it can serve as a good example of the benefits of adding OpIDs and the TIMELINE view.