Caddy iso8601

[Bit0r]

I'm using the following command to parse Caddy logs:

goaccess \
    ./logs/$remote_host/$NUXT_SITE_NAME.jsonl \
    -o ./logs/$remote_host/$NUXT_SITE_NAME.html \
    --datetime-format '%Y-%m-%dT%H:%M:%S.%f%z' \
    --log-format '{"ts":"%x","request":{"client_ip":"%h","proto":"%H","method":"%m","host":"%v","uri":"%U","headers":{"User-Agent":["%u"],"Referer":["%R"]},"tls":{"cipher_suite":"%k","proto": "%K"}},"duration": "%T","size": "%b","status": "%s","resp_headers":{"Content-Type":["%M"]}}'

log {
		output file /var/log/caddy/{$NUXT_SITE_NAME}.jsonl {
			mode 640
			roll_size 10MiB
			roll_keep 100
			roll_keep_for 90d
		}
		format json {
			time_format iso8601
		}
		level DEBUG
	}

An error occurred:

==77101==
==77101== Token '2025-01-04T19:46:44.435+0800' doesn't match specifier '%x'
==77101== Token '2025-01-04T19:46:45.234+0800' doesn't match specifier '%x'
==77101== Token '2025-01-04T19:47:44.519+0800' doesn't match specifier '%x'
==77101== Token '2025-01-04T19:47:45.294+0800' doesn't match specifier '%x'
==77101== Token '2025-01-04T19:48:44.340+0800' doesn't match specifier '%x'
==77101== Token '2025-01-04T19:48:45.152+0800' doesn't match specifier '%x'
==77101== Token '2025-01-04T19:49:44.379+0800' doesn't match specifier '%x'
==77101== Token '2025-01-04T19:49:45.164+0800' doesn't match specifier '%x'
==77101== Token '2025-01-04T19:50:44.572+0800' doesn't match specifier '%x'
==77101== Token '2025-01-04T19:50:45.338+0800' doesn't match specifier '%x'
==77101==

One of my Caddy log lines is:

{
  "level": "error",
  "ts": "2025-01-05T23:01:44.850+0800",
  "logger": "http.log.access.log0",
  "msg": "handled request",
  "request": {
    "remote_ip": "34.123.33.225",
    "remote_port": "41640",
    "client_ip": "34.123.33.225",
    "proto": "HTTP/1.1",
    "method": "GET",
    "host": "xxxx.com",
    "uri": "/",
    "headers": {
      "User-Agent": [
        "SentryUptimeBot/1.0 (+http://docs.sentry.io/product/alerts/uptime-monitoring/)"
      ],
      "Referer": [
        "http://xxxx.com/"
      ],
      "Sentry-Trace": [
        ""
      ],
      "Accept": [
        "*/*"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "",
      "server_name": "xxxx.com"
    }
  },
  "bytes_read": 0,
  "user_id": "",
  "duration": 0.000525432,
  "size": 0,
  "status": 502,
  "resp_headers": {
    "Server": [
      "Caddy"
    ],
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ]
  }
}

[Bit0r]

I think '%Y-%m-%dT%H:%M:%S.%f%z' should be able to match '2025-01-05T23:01:44.850+0800'?

[allinurl]

This is going to be tricky without preprocessing, since I don't think strptime(3) supports the %f specifier. Do you need the timezone? or would you be able to get rid of the ms part? workaround without tz:

goaccess access.log --log-format='{"ts":"%dT%t.%^","request":{"client_ip":"%h","proto":"%H","method":"%m","host":"%v","uri":"%U","headers":{"User-Agent":["%u"],"Referer":["%R"]},"tls":{"cipher_suite":"%k","proto": "%K"}},"duration": "%T","size": "%b","status": "%s","resp_headers":{"Content-Type":["%M"]}}' --date-format='%Y-%m-%d' --time-format=%T --date-spec=min

[Bit0r]

I completed the parsing using the following command:

sd '"ts":"(.{19}).{4}(.+?)"' '"ts":"$1$2"' <./logs/$remote_host/$site_name.jsonl \
    | goaccess - \
    -o ./logs/$remote_host/$site_name.html \
    -j 6 \
    --datetime-format '%FT%T%z' \
    --log-format '{"ts":"%x","request":{"client_ip":"%h","proto":"%H","method":"%m","host":"%v","uri":"%U","headers":{"User-Agent":["%u"],"Referer":["%R"]},"tls":{"cipher_suite":"%k","proto": "%K"}},"duration": "%T","size": "%b","status": "%s","resp_headers":{"Content-Type":["%M"]}}'

[Bit0r]

You might consider adding it to the usage examples of goaccess, as it is a solution.

[allinurl]

You might consider adding it to the usage examples of goaccess, as it is a solution.

Awesome, thanks for sharing that solution!

[Bit0r]

You might consider adding it to the usage examples of goaccess, as it is a solution.

Awesome, thanks for sharing that solution!

You're welcome.