Elasticsearch for Logs: Powerful, Hungry, and Often Overkill

Elasticsearch was built to search documents. Logs are not documents — they are events. That single mismatch explains almost every cost and every complaint about using Elasticsearch for log management, and it deserves a proper technical unpacking, because the trade-off is real on both sides.

An event has a timestamp, a handful of structured fields, and maybe a message string. It arrives once, is never updated, gets queried in bulk by time range and field value, and ages into irrelevance within weeks. A document — a product listing, a wiki page — is the opposite: updated in place, queried by relevance-ranked text match, valuable indefinitely.

Elasticsearch’s core data structure, the inverted index, is a marvel for the second workload. For the first, it is a tax you pay on every byte.

Inverted indexes vs columnar storage, concretely

An inverted index maps every term to the list of documents containing it. Index the line:

ERROR payment-service timeout connecting to postgres after 5000ms

and Elasticsearch tokenizes it, then updates posting lists for error, payment-service, timeout, connecting, postgres, and so on. Later, searching for postgres timeout intersects two posting lists and returns in milliseconds regardless of corpus size. This is genuinely magical, and nothing else on this page replicates it.

The costs: every token in every field gets indexed at write time (CPU), posting lists and doc values occupy disk alongside the source (storage amplification), and query-time structures compete for a JVM heap that best practice caps around 30–32GB per node. Write-heavy, read-rarely workloads — which is what logs are — pay the full write cost on 100% of data and collect the search benefit on a fraction of a percent.

A columnar store like ClickHouse inverts the bet. Data lands sorted by a primary key you choose — typically (service, timestamp) for logs. Each column is stored contiguously and compressed as a unit. A million consecutive status_code values that are mostly 200 compress to almost nothing; timestamps delta-encode beautifully; repeated service names all but vanish. Compression ratios of 10–30x on log data are routine, where Elasticsearch often stores logs at or above raw size depending on mappings. The ClickHouse docs describe the MergeTree mechanics if you want the details.

The query trade follows directly. WHERE service = 'payment' AND ts > now() - INTERVAL 1 HOUR skips irrelevant data at the block level and scans only two columns — fast and cheap. Free-text search means scanning a compressed message column, which works surprisingly well at moderate scale (decompression is fast, and skip indexes like token bloom filters help) but will never match a real inverted index on huge corpora.

So the architecture question reduces to a workload question: what fraction of your queries need text search that field filters cannot express?

Field queries cover more than you think

Be honest about your last twenty log investigations. Mine, roughly reconstructed, were:

• Filter by trace ID (field).
• Filter by service + status ≥ 500 + time window (fields).
• Filter by user ID across services (field).
• Count errors grouped by endpoint after a deploy (fields + aggregation).
• One genuine text search: hunting a stack-trace fragment across services when I did not know which one threw it.

That ratio — nineteen field queries to one text search — matches most backend teams I have worked with, and it holds because modern structured logging front-loads the work. If your applications emit JSON with consistent field names, the interesting dimensions are already extracted at write time. The text search that remains is usually “find this substring in the message,” which a columnar scan over one service’s recent data handles in seconds.

Full-text search over logs is genuinely the requirement when: your logs are unstructured and will stay that way (legacy systems, appliance syslog), security or compliance workflows demand arbitrary retroactive text discovery across long retention windows, or support workflows search by customer-visible strings that never became fields. Those are real cases. They are also the minority.

ILM: the feature that is also the bill

Elastic’s answer to log economics is Index Lifecycle Management: hot nodes on fast NVMe take writes, warm nodes hold recent read-mostly indexes, cold nodes hold shrunk and force-merged ones, frozen tiers page in searchable snapshots from object storage, and eventually indexes get deleted. As of recent versions this machinery is well-integrated and it genuinely works.

But look at what you have signed up to operate:

• Rollover policies with size and age triggers per index pattern.
• Multiple hardware profiles in one cluster, with allocation rules routing shards between them.
• Shrink, force-merge, and migrate actions that run in the background and occasionally wedge, leaving indexes stuck mid-phase for you to unstick by hand.
• Snapshot repositories and their lifecycle for the frozen tier.

ILM exists because storing logs in an inverted index is expensive enough to need a mitigation system. The mitigation system then needs its own monitoring. I have been paged for a full hot tier at 3am because a rollover alias silently broke and one index absorbed nine days of writes — the failure was not Elasticsearch malfunctioning, it was the complexity budget of the workaround coming due.

Columnar stores mostly do not need this layer. A TTL clause and a tiered-storage policy replace the entire phase machine, because the underlying storage is already cheap enough not to require choreography.

Mappings: the schema you have whether you want one or not

One more tax that surprises teams coming from schemaless expectations: Elasticsearch infers a mapping for every field it sees, and log data is a mapping-explosion machine.

Dynamic mapping means the first document to arrive with a given field decides its type. A service that logs "duration": 250 one day and "duration": "250ms" the next produces indexing failures or, depending on settings, silently dropped documents. Multiply by every team emitting JSON with slightly different conventions, and the daily index becomes a negotiation between codebases that have never met.

Then there is field count. Each mapped field carries index structures and heap footprint, and Elasticsearch caps fields per index at 1,000 by default for good reason. Logs with user-generated keys — a properties object keyed by whatever developers put in it — blow through that cap fast. The fixes (flattened field types, strict mappings, ignore_above, dropping fields at ingest) all work, and all require someone to own a schema for data that was supposed to be schemaless.

Columnar stores are not immune to schema questions — you still choose columns and types — but the failure mode is different: a decision made once at table-design time, rather than an emergent property renegotiated by every deploy across every service. For log pipelines maintained by small teams, “decided once, explicitly” beats “inferred continuously, implicitly” nearly every time.

Shard sizing, the perennial 3am topic

Shards are Elasticsearch’s unit of distribution and recovery, and log workloads stress them in a specific way: time-based indexes multiply.

Daily indexes across dozens of services, each with a replica, and within months you have thousands of shards. Every shard carries heap overhead and cluster-state weight. The long-standing guidance is to keep shards in the tens of GB and total shard count proportional to heap — and time-based log indexes violate both by default, because index creation is driven by the calendar, not by data volume. Your quiet staging services generate the same number of indexes as production, each a few MB, each carrying the same fixed overhead.

So you tune: rollover by size instead of by day, shrink old indexes, merge small ones, cap total_shards_per_node. All solvable. All ongoing. When a data node dies, recovery time is a function of shard count and size, and a cluster with poorly sized shards can spend hours rebalancing while queries degrade — which is precisely when your team most wants to query logs, because whatever killed the node probably hurt production too.

Where that leaves you

Choose Elasticsearch for log management when text search over logs is a first-class requirement, when retention and compliance workflows demand it, or when your organization already carries the expertise and the marginal cost of one more workload is low. It remains the strongest search engine in the space, and pretending otherwise helps nobody.

Skip it when your logs are structured, your queries are field-shaped, and your team is small. You would be paying the inverted-index tax — write amplification, heap pressure, ILM choreography, shard math — for a capability you exercise once a month, and a columnar or index-light store will do the daily work faster for a fraction of the hardware.

For the broader stack-level view, including where Loki and Kibana fit into the picture, see the ELK stack in 2026: still worth the memory bill? My position condensed: Elasticsearch is a search engine you can use for logs, not a log store that happens to search. Buy it for the search, or do not buy it at all.