Linux Log Management After journald Took Over

The war is over and journald won. On any mainstream distribution from the last decade, linux log management starts at the systemd journal: every unit’s stdout, syslog traffic, and kernel messages land there first, indexed and queryable, before anything else sees them. You can still bolt rsyslog on top, and plenty of shops do, but treating the journal as the primary interface — rather than an implementation detail under /var/log/messages — is the difference between fighting the system and using it.

The catch is that journald ships with defaults that quietly work against you: volatile storage on some setups, rate limiting that drops logs without much ceremony, and disk caps you discover only when old logs vanish. Here is the working knowledge, in the order you will need it.

The journalctl fluency that pays rent

Most engineers know journalctl -u nginx and stop there. The next tier of flags is where the tool starts replacing your grep habits.

Units and time windows. These compose, and the time parser is forgiving:

journalctl -u myapp.service --since "2026-05-27 09:00" --until "09:30"
journalctl -u myapp -u postgresql --since -2h     # two units, last two hours
journalctl -b -1 -u myapp                          # previous boot — post-mortem gold

That -b -1 deserves emphasis. When a box rebooted unexpectedly, the previous boot’s journal is your flight recorder — assuming storage is persistent, which we will get to.

Priorities. Syslog levels survive into the journal. -p err shows err and worse; ranges work too:

journalctl -p err --since today          # everything at err, crit, alert, emerg
journalctl -p warning..err -u myapp      # a band of severities

Running journalctl -p err -b on a misbehaving host before anything else is a habit that has shortened more incidents for me than any dashboard.

Structured output. Every journal entry carries metadata fields — unit, PID, UID, boot ID, and anything the app attached. -o json exposes it all, which turns journalctl into a data source for scripts:

journalctl -u myapp -o json --since -1h | jq -r 'select(._PID) | ._PID' | sort | uniq -c
journalctl -o verbose -n 5              # see every field on recent entries

Also worth knowing: journalctl -f for tailing, -k for kernel messages only, and _COMM=sshd style field matches for filtering without a unit.

Persistence: the default you must not trust

Whether the journal survives a reboot depends on whether /var/log/journal/ exists. With Storage=auto — a common default — journald writes to persistent storage only if that directory is present; otherwise everything lives in /run/log/journal, which is tmpfs, which means a reboot erases your evidence. Some distributions create the directory for you; some do not. This is exactly the kind of variance you want to eliminate by declaring intent:

# /etc/systemd/journald.conf
[Journal]
Storage=persistent

Then systemctl restart systemd-journald. Verify with journalctl --list-boots — if you see only the current boot on a machine that has rebooted before, you have been running volatile and every past incident’s logs are already gone.

I learned this one the annoying way: a server that kernel-panicked nightly for a week, and every morning the journal began at boot. The fix took thirty seconds. Finding out I needed it took seven days.

Rate limiting drops logs silently

journald protects itself from log floods with per-service rate limiting — by default on the order of a burst allowance within a short interval (the exact numbers have shifted between versions; check man journald.conf on your systems). When a service exceeds it, journald drops the excess and writes a single line noting how many messages were suppressed.

That design is sane. The failure mode is not: the service most likely to blow through the limit is one erroring in a tight loop — precisely when you want every line. And the suppression notice is easy to miss because it appears once, attributed to journald, not interleaved as gaps in your app’s output.

Two defenses. First, actually watch for suppression:

journalctl -u systemd-journald | grep -i suppressed

If that grep returns rows for a service you were debugging, your logs from that window have holes. Second, for services where completeness matters more than flood protection, raise or disable the limit — either globally in journald.conf or per-unit:

# in the service unit, [Service] section
LogRateLimitIntervalSec=0

Zero disables it for that unit. Use sparingly and deliberately; the protection exists because a runaway logger can eat a disk.

Disk usage: SystemMaxUse and friends

Persistent journals grow. journald caps them by default at a percentage of the filesystem, but on a small VPS “a percentage of the disk” can still crowd out your actual workload, and the default vacuuming silently deletes your oldest logs to stay under the cap. Set explicit numbers so the trade-off is one you chose:

[Journal]
SystemMaxUse=2G        # total journal size cap
SystemKeepFree=1G      # always leave this much disk free
MaxRetentionSec=1month # age-based expiry, if you want one

For immediate cleanup, journalctl --vacuum-size=1G or --vacuum-time=2weeks prune on demand, and journalctl --disk-usage tells you where you stand. Roughly: a moderately busy single service produces tens to a few hundred MB of journal per day, so 2G buys you days to weeks of local history — which is the right way to think about the local journal anyway: a buffer, not an archive.

Forwarding: the journal is a buffer, not a destination

One host’s journal answers questions about one host. The moment you have three servers, or the moment a disk dies taking its journal with it, you need logs somewhere central. The clean patterns, in rough order of preference:

• A collector reading the journal directly. Vector, Fluent Bit, and similar agents have journald sources that preserve all the structured fields and ship them wherever you aggregate — over HTTP or OTLP to your log platform. This is the modern default and what I would set up today.
• systemd-journal-upload to a systemd-journal-remote receiver — journal-native, a bit spartan, fine for small fleets that want zero third-party agents.
• ForwardToSyslog=yes into rsyslog with a remote target — the traditional route, still perfectly serviceable if rsyslog is already in the picture.

Whichever you choose, forwarding changes the security posture: your logs now traverse a network and rest on another machine, and they contain more than you think (paths, usernames, sometimes worse). Encrypt the transport and think about who can read the destination — securing your logs covers that side properly.

The end state to aim for is boring: Storage=persistent, explicit size caps, rate limits you have consciously accepted, an agent forwarding to central storage, and a team that reaches for journalctl -p err -b before anything else. None of it is glamorous. All of it is the difference between having the evidence and having a story about how you lost it.