A fast and powerful log viewer and processor that translates JSON or logfmt logs into a pretty human-readable format. High performance and convenient features are the main goals.
- Automatic usage of the less pager by default for convenience.
- Log streaming with the
-P
flag that disables the pager. - Log record filtering by field key/value pairs with the
-f
option with support for hierarchical keys. - Quick and easy filtering by level with the
-l
option. - Quick and easy filtering by timestamp range using the
--since
and--until
options and intuitive formats:- RFC-3339 timestamp format.
- Current configured timestamp output format with the
-t
option or environment variable. - Human friendly shortcuts like
today
,yesterday
,friday
or relative offsets like-3h
or-14d
.
- Quick and easy hiding and revealing of fields with the
-h
option. - Hide empty fields with the
-e
flag. - Lightning fast message sorting with automatic indexing for local files using the
-s
flag.- Handles ~1 GiB/s for the first scan and allows fast filtering by timestamp range and level without scanning the data afterwards.
- Works fast with hundreds of local files containing hundreds of gigabytes of data.
- Reindexes large, growing files at lightning speed, skipping unmodified blocks, ~10 GiB/s.
- Follow mode with live message sorting by timestamp from different sources using the
-F
flag and preview of several recent messages with the--tail
option. - Custom complex queries that can include and/or conditions and much more.
- Non-JSON prefixes with
--allow-prefix
flag. - Displays timestamps in UTC by default and supports easy timezone switching with the
-Z
option and the-L
flag for a local timezone. - Customizable via configuration file and environment variables, supports easy theme switching and custom themes.
- See performance section for more details.
-
Install using homebrew on macOS or Linux
brew install pamburus/tap/hl
-
Download and extract using
curl
andtar
on macOScurl -sSfL https://github.com/pamburus/hl/releases/latest/download/hl-macos.tar.gz | tar xz
-
Download and extract using
curl
andtar
on Linuxcurl -sSfL https://github.com/pamburus/hl/releases/latest/download/hl-linux-x86_64-musl.tar.gz | tar xz
-
Install AUR package on Arch Linux
yay -S hl-log-viewer-bin
-
Install using cargo
cargo install --locked --git https://github.com/pamburus/hl.git
-
Download latest release from download page
See other screenshots
-
Concatenate all log files
Command
hl *.log
Concatenates and displays all
*.log
files found in the current directory.
-
Concatenate all log files including compressed log files
Command
hl $(ls -tr /var/log/example/*.{log,log.gz,log.zst,s})
Concatenates and displays all
*.log
,*.log.gz
,*.log.zst
and*.s
(will detect compression) files found in/var/log/example/
.
-
Use the default pager with the default parameters
Command
hl example.log
Automatically opens
less
pager with the default parameters. -
Override options for default pager
Command
LESS=-SR hl example.log
Opens
less
pager with disabled line wrapping. -
Use custom pager
Command
PAGER="most -w" hl example.log
Opens
most
pager with-w
option.
-
Errors only
Command
hl -l e
Displays only error log level messages.
-
Errors and warnings
Command
hl -l w
Displays only warning and error log level messages.
-
Errors, warnings and informational
Command
hl -l i
Displays all log messages except debug level messages.
-
Command
tail -f example.log | hl -P
Tracks changes in the example.log file and displays them immediately. Flag
-P
disables automatic using of pager in this case.
-
Command
hl example.log --filter component=tsdb
Displays only messages where the
component
field has the valuetsdb
. -
Command
hl example.log -f component!=tsdb -f component!=uninteresting
Displays only messages where the
component
field has a value other thantsdb
oruninteresting
. -
Command
hl example.log -f provider~=string
Displays only messages where the
provider
field contains thestring
sub-string. -
Command
hl example.log -f 'provider!~=string'
Displays only messages where the
provider
field does not contain thestring
sub-string.
-
Command
hl my-service.log --query 'level > info or status-code >= 400 or duration > 0.5'
Displays messages that either have a level higher than info (i.e. warning or error) or have a status code field with a numeric value >= 400 or a duration field with a numeric value >= 0.5.
-
Command
hl my-service.log -q '(request in (95c72499d9ec, 9697f7aa134f, bc3451d0ad60)) or (method != GET)'
Displays all messages that have the 'request' field with one of these values, or the 'method' field with a value other than 'GET'.
-
Complete set of supported operators
- Logical operators
- Logical conjunction -
and
,&&
- Logical disjunction -
or
,||
- Logical negation -
not
,!
- Logical conjunction -
- Comparison operators
- Equal -
eq
,=
- Not equal -
ne
,!=
- Greater than -
gt
,>
- Greater or equal -
ge
,>=
- Less than -
lt
,<
- Less or equal -
le
,<=
- Equal -
- String matching operators
- Sub-string check - (
contain
,~=
), (not contain
,!~=
) - Wildcard match - (
like
), (not like
)- Wildcard characters are:
*
for zero or more characters and?
for a single character
- Wildcard characters are:
- Regular expression match - (
match
,~~=
), (not match
,!~~=
)
- Sub-string check - (
- Operators with sets
- Test if value is one of the values in a set -
in (v1, v2)
,not in (v1, v2)
- Test if value is one of the values in a set loaded from a file -
in @filename
,not in @filename
, assuming that each element is a line in the file, which can be either a simple string or a JSON string - Test if value is one of the values in a set loaded stdin -
in @-
,not in @-
- Test if value is one of the values in a set -
- Logical operators
-
Notes
-
Special field names that are reserved for filtering by predefined fields regardless of the actual source field names used to load the corresponding value:
level
,message
,caller
andlogger
. -
To address a source field with one of these names instead of predefined fields, add a period before its name, i.e.,
.level
will perform a match against the "level" source field. -
To address a source field by its exact name, use a JSON-formatted string, i.e.
-q '".level" = info'
. -
To specify special characters in field values, also use a JSON-formatted string, i.e.
hl my-service.log -q 'message contain "Error:\nSomething unexpected happened"'
-
-
Command
hl example.log --since 'Jun 19 11:22:33' --until yesterday
Displays only messages that occurred after Jun 19 11:22:33 UTC of the current year (or the previous year if the current date is less than Jun 19 11:22:33) and before yesterday midnight.
-
Command
hl example.log --since -3d
Displays only messages from the past 72 hours.
-
Command
hl example.log --until '2021-06-01 18:00:00' --local
Displays only messages that occurred before 6 PM local time on June 1, 2021, and shows timestamps in local time.
-
Command
hl example.log --hide provider
Hides field
provider
. -
Command
hl example.log --hide '*' --hide '!provider'
Hides all fields except
provider
. -
Command
hl example.log -h headers -h body -h '!headers.content-type'
Hides fields
headers
andbody
but shows a single sub-fieldcontent-type
inside fieldheaders
.
-
Command
hl -s *.log
Displays log messages from all log files in the current directory sorted in chronological order.
-
Command
hl --sync-interval-ms 500 -F <(kubectl logs -l app=my-app-1 -f) <(kubectl logs -l app=my-app-2 -f)
Runs without a pager in follow mode by merging messages from the outputs of these 2 commands and sorting them chronologically within a custom 500ms interval.
-
Command
hl -F --tail 100 app1.log app2.log app3.log
Runs without a pager in follow mode, following the changes in three log files in the current directory and sorting them chronologically at a default interval of 100ms. Preloads 100 lines from the end of each file before filtering.
-
Configuration file is automatically loaded if found in a predefined platform-specific location.
OS Location macOS ~/.config/hl/config.{yaml,toml,json} Linux ~/.config/hl/config.{yaml,toml,json} Windows %USERPROFILE%\AppData\Roaming\hl\config.{yaml,toml,json} -
The path to the configuration file can be overridden using the HL_CONFIG environment variable.
-
All parameters in the configuration file are optional and can be omitted. In this case, default values are used.
- Many parameters that are defined in command line arguments and configuration files can also be specified by environment variables.
- Configuration file
- Environment variables
- Command-line arguments
HL_TIME_FORMAT='%y-%m-%d %T.%3N'
overrides the time format specified in the configuration file.HL_TIME_ZONE=Europe/Berlin
overrides the time zone specified in the configuration file.HL_CONCURRENCY=4
overrides the concurrency limit specified in the configuration file.HL_PAGING=never
specifies the default value for the paging option, but it can be overridden by command line arguments.
- Using
theme
value in the configuration file. - Using environment variable, i.e.
HL_THEME=classic
, overrides the value specified in configuration file. - Using command-line argument, i.e.
--theme classic
, overrides all other values.
To select themes with preview fzf tool can be used like this:
hl --list-themes | fzf --preview-window="top,80%" --preview="head -n 100 example.log | hl -c --theme {}"
-
Custom themes are automatically loaded when found in a predefined platform-specific location.
OS Location macOS ~/.config/hl/themes/*.{yaml,toml,json} Linux ~/.config/hl/themes/*.{yaml,toml,json} Windows %USERPROFILE%\AppData\Roaming\hl\themes*.{yaml,toml,json} -
Format description
-
Section
elements
contains styles for predefined elements. -
Section
levels
contains optional overrides for styles defined inelements
sections per logging level, which are [debug
,info
,warning
,error
]. -
Each element style contains optional
background
,foreground
andmodes
parameters. -
Example
elements: <element>: foreground: <color> background: <color> modes: [<mode>, <mode>, ...] levels: <level>: <element>: foreground: <color> background: <color> modes: [<mode>, <mode>, ...]
-
Color format is one of
- Keyword
default
specifies default color defined by the terminal. - ASCII basic color name, one of
black
red
green
yellow
blue
magenta
cyan
white
bright-black
bright-red
bright-green
bright-yellow
bright-blue
bright-magenta
bright-cyan
bright-white
- 256-color palette code, from
0
to255
. - RGB color in hex web color format, i.e.
#FFFF00
for bright yellow color.
- Keyword
-
Modes is a list of additional styles, each of them is one of
bold
faint
italic
underline
slow-blink
rapid-blink
reverse
conceal
crossed-out
-
- One Dark Neo
- Built-in "Light Background" color scheme
- One Dark Neo
- Note: It is recommended to use
draw_bold_text_with_bright_colors: true
setting
- Note: It is recommended to use
- Light
- Note: It is recommended to use
draw_bold_text_with_bright_colors: false
setting
- Note: It is recommended to use
JSON and logfmt log converter to human readable representation
Usage: hl [OPTIONS] [FILE]...
Arguments:
[FILE]... Files to process
Options:
--config <FILE> Configuration file path [env: HL_CONFIG=]
-s, --sort Sort messages chronologically
-F, --follow Follow input streams and sort messages chronologically during time frame set by --sync-interval-ms option
--tail <N> Number of last messages to preload from each file in --follow mode [default: 10]
--sync-interval-ms <MILLISECONDS> Synchronization interval for live streaming mode enabled by --follow option [default: 100]
--paging <WHEN> Control pager usage (HL_PAGER or PAGER) [env: HL_PAGING=] [default: auto] [possible values: auto, always, never]
-P Handful alias for --paging=never, overrides --paging option
--help Print help
-V, --version Print version
Filtering Options:
-l, --level <LEVEL> Filter messages by level [env: HL_LEVEL=]
--since <TIME> Filter messages by timestamp >= <TIME> (--time-zone and --local options are honored)
--until <TIME> Filter messages by timestamp <= <TIME> (--time-zone and --local options are honored)
-f, --filter <FILTER> Filter messages by field values [k=v, k~=v, k~~=v, 'k!=v', 'k!~=v', 'k!~~=v'] where ~ does substring match and ~~ does regular expression match
-q, --query <QUERY> Filter using query, accepts expressions from --filter and supports '(', ')', 'and', 'or', 'not', 'in', 'contain', 'like', '<', '>', '<=', '>=', etc
Output Options:
--color [<WHEN>] Color output control [env: HL_COLOR=] [default: auto] [possible values: auto, always, never]
-c Handful alias for --color=always, overrides --color option
--theme <THEME> Color theme [env: HL_THEME=] [default: universal]
-r, --raw Output raw source messages instead of formatted messages, which can be useful for applying filters and saving results in their original format
--no-raw Disable raw source messages output, overrides --raw option
--raw-fields Output field values as is, without unescaping or prettifying
-h, --hide <KEY> Hide or reveal fields with the specified keys, prefix with ! to reveal, specify '!*' to reveal all
--flatten <WHEN> Whether to flatten objects [env: HL_FLATTEN=] [default: always] [possible values: never, always]
-t, --time-format <FORMAT> Time format, see https://man7.org/linux/man-pages/man1/date.1.html [env: HL_TIME_FORMAT=] [default: "%b %d %T.%3N"]
-Z, --time-zone <TZ> Time zone name, see column "TZ identifier" at https://en.wikipedia.org/wiki/List_of_tz_database_time_zones [env: HL_TIME_ZONE=] [default: UTC]
-L, --local Use local time zone, overrides --time-zone option
--no-local Disable local time zone, overrides --local option
-e, --hide-empty-fields Hide empty fields, applies for null, string, object and array fields only [env: HL_HIDE_EMPTY_FIELDS=]
-E, --show-empty-fields Show empty fields, overrides --hide-empty-fields option [env: HL_SHOW_EMPTY_FIELDS=]
--input-info <VARIANT> Show input number and/or input filename before each message [default: auto] [possible values: auto, none, full, compact, minimal]
-o, --output <FILE> Output file
Input Options:
--input-format <FORMAT> Input format [env: HL_INPUT_FORMAT=] [default: auto] [possible values: auto, json, logfmt]
--unix-timestamp-unit <UNIT> Unix timestamp unit [env: HL_UNIX_TIMESTAMP_UNIT=] [default: auto] [possible values: auto, s, ms, us, ns]
--allow-prefix Allow non-JSON prefixes before JSON messages [env: HL_ALLOW_PREFIX=]
--delimiter <DELIMITER> Log message delimiter, [NUL, CR, LF, CRLF] or any custom string
Advanced Options:
--interrupt-ignore-count <N> Number of interrupts to ignore, i.e. Ctrl-C (SIGINT) [env: HL_INTERRUPT_IGNORE_COUNT=] [default: 3]
--buffer-size <SIZE> Buffer size [env: HL_BUFFER_SIZE=] [default: "256 KiB"]
--max-message-size <SIZE> Maximum message size [env: HL_MAX_MESSAGE_SIZE=] [default: "64 MiB"]
-C, --concurrency <N> Number of processing threads [env: HL_CONCURRENCY=]
--shell-completions <SHELL> Print shell auto-completion script and exit [possible values: bash, elvish, fish, powershell, zsh]
--man-page Print man page and exit
--list-themes Print available themes and exit
--dump-index Print debug index metadata (in --sort mode) and exit
--debug Print debug error messages that can help with troubleshooting
- MacBook Pro (16-inch, 2021)
-
CPU: Apple M1 Max CPU
-
OS: macOS Sonoma 14.4.1
-
Data: ~ 2.3 GiB log file, 6 000 000 lines
-
hl v0.28.0 ~ 1.4 seconds
$ time hl example.log -c -o /dev/null hl example.log -c -o /dev/null 11.74s user 0.53s system 885% cpu 1.386 total
-
hlogf v1.4.1 ~ 8.5 seconds
$ time hlogf example.log --color always >/dev/null hlogf example.log --color always > /dev/null 6.93s user 1.79s system 99% cpu 8.757 total
-
humanlog v0.7.6 ~ 77 seconds
$ time humanlog <example.log --color always >/dev/null humanlog> reading stdin... humanlog --color always < example.log > /dev/null 80.02s user 4.71s system 109% cpu 1:17.11 total
-
fblog v4.9.0 ~ 36 seconds
$ time fblog example.log >/dev/null fblog example.log > /dev/null 32.48s user 2.03s system 97% cpu 35.526 total
-
fblog with
-d
flag v4.9.0 ~ 148 seconds$ time fblog -d example.log >/dev/null fblog -d example.log > /dev/null 132.12s user 14.39s system 99% cpu 2:27.61 total
-
-
See #132 for how to repeat measurements
-