Kubernetes Log Management for Clusters Nobody Officially Owns

There is a specific kind of cluster this article is for: three to ten nodes, set up by someone who may have since left, running production workloads, owned by “the backend team” in the sense that nobody else will admit to it. Kubernetes log management for that cluster cannot be a platform project, because there is no platform team. It has to be a small, boring setup that one person can install in an afternoon and then mostly forget.

Here is that setup. Four rules, one agent, one config sketch, and a list of things you are explicitly allowed to skip.

Rule 1: everything logs to stdout, no exceptions

Kubernetes has one blessed logging path: containers write to stdout/stderr, the runtime writes those streams to files on the node (under /var/log/pods/, per the Kubernetes docs), and anything on the node can read them.

Every deviation from this path — apps writing to files inside the container, custom log volumes, in-process shipping to a remote endpoint — creates a snowflake you will forget about until the day its logs are missing. I have debugged an outage where the one service that “helpfully” wrote to its own logfile inside an emptyDir was, of course, the service that crashed, taking its logs with it.

So the rule is absolute: stdout, structured JSON if you can get it, one event per line. If a third-party container insists on writing files, run a minimal tail-to-stdout arrangement for that one pod and treat it as tech debt. This is the only rule that requires changing application behavior, and it is worth every argument it causes.

Rule 2: one DaemonSet, not a sidecar zoo

Collection is a node problem, not a pod problem. Run a single lightweight agent as a DaemonSet — one pod per node, reading /var/log/pods/, enriching with pod metadata, shipping everything to one central endpoint. Fluent Bit and Vector are both solid choices here; both are small (tens of MB of memory per node in typical setups), both speak every major output protocol.

Here is a trimmed Fluent Bit configuration as a sketch — treat it as a starting point to check against current docs, not something to paste blind, since input names and defaults shift between versions:

[INPUT]
    Name              tail
    Path              /var/log/pods/*/*/*.log
    Parser            cri
    Tag               kube.*
    Mem_Buf_Limit     16MB
    Skip_Long_Lines   On

[FILTER]
    Name              kubernetes
    Match             kube.*
    Merge_Log         On          # parse JSON in the log field
    Keep_Log          Off

[OUTPUT]
    Name              http
    Match             kube.*
    Host              logs.internal.example.com
    Port              443
    tls               On
    Format            json
    Retry_Limit       5

The kubernetes filter is the part that earns its keep: it attaches namespace, pod name, container name, and labels to every line, which is what makes the next rule possible.

Set memory limits on the agent and set them honestly. An unbounded log shipper on a node with a misbehaving chatty pod becomes the second incident of the night.

Rule 3: route by labels, not by parsing

You already label your workloads (or should): app, team, env. Since the DaemonSet agent attaches those labels to every log line, all your routing decisions become label matches — no regex on message bodies, no per-service parser configs.

Practical routing for a small cluster is short:

That last row matters more than it looks. In most small clusters, the ingress controller’s access logs are half the total volume and answer questions your app logs already answer. Dropping or sampling them at the agent is the single cheapest cost cut available, and it costs one filter stanza.

Where should the central endpoint point? Anything that accepts JSON over HTTP or OTLP and lets you filter by those labels — Loki, a ClickHouse-backed store, a hosted service. If you are already running an observability stack like LogReplay for errors and session replay, pointing the agent at its log ingest means one less system to keep alive, which is the entire spirit of this exercise. The specifics of picking the destination are a separate decision — start here if you are centralizing logs for the first time.

Rule 4: capture the death rattle

The log that matters most is the one printed right before a container died, and there are two ways to lose it.

First: the crash loop. A pod restarts, and naive tooling shows you the fresh container’s logs — which are empty, because it just started. The previous container’s logs are still on the node; kubectl logs --previous reads them. But kubelet only keeps a limited number of rotated log files per container, so a pod crash-looping every ten seconds churns through that window fast. This is exactly why the DaemonSet must ship continuously: by the time you are investigating, the node-local files may be gone, but the central store has every line from every incarnation.

Second: rotation on a chatty node. Kubelet rotates container logs at a size threshold (configurable via containerLogMaxSize and containerLogMaxFiles; check your distribution’s defaults — managed offerings differ). If a pod logs faster than your agent tails during an incident, rotation can outrun collection. The Mem_Buf_Limit and file-buffer settings in your agent are the defense; make sure the agent’s own “I am dropping data” metrics go somewhere a human will see.

While you are at it, alert on one more thing: the DaemonSet itself missing from any node. A logging agent that silently isn’t running is worse than none, because you believe you have coverage.

What you are allowed to skip

Everything else, basically. Specifically and with confidence:

• Sidecar log shippers per pod. They multiply memory overhead by your pod count and configuration by your service count. The node-level agent already sees everything.
• A log operator or CRD-based routing framework. These earn their complexity at platform-team scale, with dozens of teams and tenant isolation requirements. You have neither.
• In-cluster log storage. Running your log store inside the cluster it monitors means losing your logs at the exact moment you need them. Ship out — to a managed store, a separate VM, anywhere with an independent failure domain.
• Parsing pipelines beyond JSON. If a service’s logs need custom regex parsing, fix the service to emit JSON. Parsing configs rot; log formats drift; structured output at the source does not.

The whole setup is one DaemonSet manifest, one config, and a labeling convention you mostly had already. It survives node failures, catches crash loops, and needs attention roughly never — which is the correct amount for a cluster nobody officially owns.