Centralizing Logs When Every Service Writes to Its Own File

You have six services on four boxes, each appending to its own file, and debugging a cross-service request means four SSH sessions and a lot of grep. The fix is centralized log management, but here’s the mistake I watch teams make over and over: they treat it as one big project — pick Elasticsearch or Loki, design the schema, migrate everything — and the project dies in planning, and eight months later they’re still grepping. Centralization is not one migration. It’s four small ones, each independently useful, each shippable in under a week. Do them in order and stop at whichever stage your team’s pain actually ends.

Stage 0: admit what SSH-and-grep costs you

Don’t skip the diagnosis. Grep-over-SSH has real virtues — zero infrastructure, zero cost, everyone knows it — and if you’re two services on one box, it’s honestly fine. It breaks down at three specific points, and you should confirm you’re hitting at least one before building anything:

• Cross-service requests. One user action touches three services; correlating it means matching timestamps across machines whose clocks may disagree by seconds.
• Dead machines. The logs explaining why a box died were on the box.
• Rotation. The evidence for last Tuesday’s incident was rotated and deleted before anyone looked.

If none of those bite yet, stop reading and go build product features. Premature observability infrastructure is still premature infrastructure.

Stage 1: one box, one agent, files as they are

The whole trick of stage 1 is changing nothing about how services log. They keep writing their files, in whatever inconsistent formats they already use. You add one agent per machine that tails the files and ships lines to a single aggregation box, which writes them to disk, organized by service and date.

That’s it. No parsing, no schema, no query engine. And yet: cross-service grep becomes one SSH session instead of four, dead machines stop taking their logs with them, and retention becomes a property of one box’s disk instead of every box’s logrotate config. Roughly 70% of the pain, gone in a day.

For the agent I’d use vector — a single static binary, small resource footprint, config in TOML. A stage-1 shipper config is genuinely this short:

# vector.toml — on each service box
[sources.service_logs]
type = "file"
include = ["/var/log/myapp/*.log"]

[sinks.aggregator]
type = "vector"
inputs = ["service_logs"]
address = "logs.internal:9000"

On the aggregation box, a matching vector source and a file sink templated by service name and date. Plain rsyslog forwarding does the same job if it’s already on your machines and you’d rather not add a binary — the architecture matters, not the agent.

One decision worth making now: give the aggregation box a bigger disk than feels necessary. Log volume only grows, and a cheap 1TB volume defers your first capacity conversation by a year.

Stage 2: structured JSON, then a real store

Two changes, in this order.

First, structure at the source. Switch services to emitting JSON lines — one object per line, with at minimum a timestamp, level, service name, message, and a request ID that’s propagated across service calls. That request ID is the single highest-value field in this entire article: it’s what turns “grep four files and squint at timestamps” into “give me every line for request X.” Most logging libraries emit JSON with a config flag; the propagation of the request ID through your HTTP clients is the actual work, and it’s a day or two.

Do this before adopting a log database, not after. Structured logs make every downstream tool better, and they make grep better too — jq on JSON lines is a shockingly capable query engine for a stage-2 team.

Then, when grep-on-the-aggregator gets slow or the team wants dashboards, add a store. For a small team the realistic candidates are Loki (cheap storage, label-based, pairs with Grafana), or a hosted service if you’d rather spend money than operations time. I’d argue against full Elasticsearch at this stage — its operational appetite is out of proportion to a small team’s needs. Your vector config grows a sink; the sources don’t change. That’s the payoff of doing stage 1 first: every later stage is a config edit on one pipeline, not a re-instrumentation of every service.

Stage 3: retention tiers, because you’ll want the history you can’t afford

Eventually someone asks for “the logs from that incident in March” and someone else asks why the log store costs what it costs. The answer to both is tiering, and the shape is standard: recent logs (a week or two) hot and queryable, a few months warm in the store with reduced detail, and everything older compressed to object storage where a terabyte costs pocket change per month. Old logs are almost never queried, but when they’re needed — an incident post-mortem, a compliance question, a “when did this behavior start” investigation — nothing substitutes. Compressed JSON on S3-class storage is so cheap that “keep everything for a year, cold” is a defensible default for a small team.

Set the tiers up when the storage bill first makes someone blink, not before.

What to deliberately defer

Half the value of the staged approach is the list of things you’re allowed to ignore. Log-based alerting: defer until your logs are structured, or you’ll build regex alerts you’ll rewrite. Dashboards: defer until someone asks the same question three times. Schema standardization committees, sampling policies, a message-queue buffer in front of ingestion: defer indefinitely; these solve volume problems you don’t have. Centralized log management for a small team is a pipeline you improve incrementally, and the teams that succeed at it are the ones that shipped stage 1 on a Tuesday instead of designing stage 4 in a document. There’s a broader set of habits that keep the pipeline useful once it exists — I’ve collected those in log management practices for small teams — but none of them matter until the logs are in one place.