Self-Hosting a Highlight-Based Stack: What the Docs Don't Say

I run one of these stacks, so consider this the notes I wish someone had handed me. The Highlight.io self hosted architecture — which LogReplay forks and ships, so this applies equally there — puts four stateful services on one box: ClickHouse for events, Kafka for ingest buffering, Postgres for application data, Redis for caching, all fronted by a Go backend. The docker compose comes up in minutes and works immediately, which is exactly the trap. Day one is easy. This post is about day ninety.

First, the honest framing: four databases on one machine is a lot of machinery for a small team. Each piece is boring, proven technology individually — that is the good news, and it is genuinely good. The operational question is whether you have the attention budget for all four at once.

What the box actually demands

The stated minimum is around 8GB RAM and 4 cores. That is real, but read it as a floor for trying it out, not a plan. ClickHouse alone is happiest with several GB to work with, Kafka’s JVM wants its share, and the collector plus backend plus Postgres plus Redis fill in the rest. On 8GB the stack runs; it just runs with no slack, and no slack means the first traffic spike or fat ClickHouse merge takes you into swap or an OOM kill. If this is production, 16GB is where the anxiety stops.

The resource that actually governs your life, though, is disk — both size and speed. Session payloads go to object storage, which is a mercy, but ClickHouse parts, Kafka log segments, and Postgres all live on local disk, and ClickHouse merges plus Kafka’s sequential writes will make a cheap network volume weep. Give it real SSD-class IOPS. And give it headroom: event data only accumulates, and (as of recent versions) the retention/TTL behavior you get out of the box may be more generous than your disk is. Decide your retention on purpose, in week one, not when the volume hits 90%.

Why the architecture holds up as well as it does on a single node is mostly a ClickHouse story — session events compress absurdly well in columnar storage, which is why every replay vendor ended up on it. You are getting a lot of storage efficiency for your operational trouble.

The two graphs that predict every incident

If you monitor nothing else, monitor these. In my experience every problem this stack has produced announced itself in one of them first.

Disk growth rate, not disk usage. A usage alert at 85% tells you that you are already in trouble. What you want is the derivative: GB per day, watched weekly. A traffic bump, a new SDK integration someone shipped, a chatty console.log left in production — all show up as a slope change days before they show up as a full disk. ClickHouse on a full volume is a bad morning; Kafka on a full volume can wedge ingest entirely.

Kafka consumer lag. This is the stack’s honest health signal. The pipeline is producer → Kafka → workers → ClickHouse, so lag is the first thing that moves when anything downstream slows: ClickHouse merges backing up, an undersized worker, a slow disk. Steady lag near zero, all is well. Growing lag with normal traffic means something downstream is sick, and you have however long the retention window is to fix it before data starts aging out unconsumed. A flat dashboard hiding a climbing lag graph is how you lose an afternoon of sessions without a single container “failing.”

Secondary tier, still worth having: ClickHouse merge/parts health, Postgres connection count, Redis memory, and a synthetic check that actually records and retrieves a session end-to-end — because “all containers green” and “the product works” are different claims.

Upgrades: the discipline part

Upgrading a multi-database compose stack is where casualness gets punished. My rules, learned the usual way:

• Snapshot first, every time. A VM or volume snapshot before any upgrade turns a catastrophe into a rollback. Postgres dumps and ClickHouse backups are nice; a whole-box snapshot is nicer.
• Read the diff between compose files. Upstream occasionally bumps ClickHouse or Kafka majors along with the app version. Application containers are stateless and forgiving; datastore version jumps are not. Know which kind of upgrade you are performing before you type the command.
• Do not skip far ahead. Sequential-ish upgrades keep any migrations working with schemas they expect. Jumping many releases at once on a stack with four datastores is how you end up reading migration source code at 2am.
• Watch Kafka lag after. It is your post-upgrade smoke test: if workers came back healthy, lag drains; if a schema migration half-applied, lag climbs while everything looks green.

Budget a calm hour per upgrade, monthly-ish. Not because it usually takes an hour — because the one time it does, you want it not to be 6pm Friday.

When hosted is honestly the better call

Total the real cost: a box with 16GB and fast SSD, plus the slice of an engineer who owns upgrades, monitoring, and the occasional weird morning. Even a modest accounting lands in the low-to-mid hundreds of dollars a month. Hosted plans for this category run from free tiers through roughly $19–249/month depending on volume. For a lot of small teams the math is not close, and picking hosted is not a failure of engineering nerve — it is reading the bill correctly.

Self-host when data residency or compliance genuinely requires it, when your event volume makes hosted pricing hurt, or when you have real ops appetite and want the control. Otherwise, take the version where Kafka lag is someone else’s graph. The stack is well-built and honestly documented; running it is a real job, and the docs cannot tell you whether you want that job. That part is on you.