Your Logs Are a Breach Waiting to Happen

A few years ago I watched an incident review where the root cause of an account takeover was not a vulnerability in the application. It was a log line. A middleware author had logged the full request object during a debugging sprint two years earlier, so every Authorization: Bearer ... header from every API call sat in plaintext in a log platform half the company could search. Nobody had rotated those tokens. The attacker who got read access to the log system didn’t need to attack anything else.

That pattern — the log system as the softest copy of your hardest secrets — is the core problem of log management security, and it gets a fraction of the attention that application security gets. Your production database has encryption, access controls, and an audit trail. Your logs contain much of the same data and frequently have none of the three.

The classic leak classes

Secrets don’t wander into logs randomly. They arrive through a small number of well-worn doors, and you can check every one of them this week.

Auth headers and cookies. The single biggest offender. Any “log the full request” middleware, any HTTP client with debug-level wire logging enabled, any reverse proxy configured to log all headers. Bearer tokens, session cookies, API keys in X-Api-Key — captured wholesale. The fix is a denylist of header names redacted before the log call is made, and the discipline to never ship curl -v-equivalent logging to production.

Query strings. URLs get logged by default at every layer — app, proxy, load balancer, CDN — and anything in the query string goes with them. Password-reset tokens, signed URLs, SSO assertions, and ?token= parameters are the classics. This is also why GET-with-secrets is an anti-pattern independent of logging: URLs leak through Referer headers and browser history too. If a secret must travel in a URL, it should be single-use and short-lived, because you should assume it will be logged somewhere you forgot about.

Stack traces with environment dumps. Some frameworks helpfully attach local variables, request context, or the entire environment to uncaught exception reports. An env dump is a credential dump: database URLs with embedded passwords, cloud provider keys, signing secrets. Crash handlers deserve the same redaction review as request loggers, and they almost never get it.

Application payloads. The developer who logs payload: %s for a user-signup event just logged an email address and possibly a password. Structured logging makes this better and worse: better because fields are enumerable and redactable by name, worse because serializing a whole object is one line of code.

Error messages from dependencies. Database drivers that echo the failed connection string, HTTP clients that include the request (with headers) in the exception text. You didn’t log the secret; your library did, when it failed.

Redaction at the source vs. at ingest

There are two places to strip sensitive data, and teams reflexively pick the wrong one.

Ingest-time redaction — regex filters in your log pipeline that scrub things matching token or card-number patterns — feels attractive because it’s centralized. One config, all services covered. But it has a structural flaw: the secret has already left the application, crossed the network, and possibly been buffered to disk by a shipper agent before the filter runs. Local files, dead-letter queues, and the shipper’s own debug output all sit upstream of your scrubber. Pattern matching also misses anything it doesn’t have a pattern for, which is most application-specific secrets.

Source-time redaction — the logging library refuses to emit sensitive fields in the first place — is the correct primary control. Structured logging helps enormously here: when logs are key-value rather than interpolated strings, you can register a denylist (password, token, authorization, secret, cookie, ssn) that the serializer redacts on every call, in every service, with no per-callsite vigilance required.

const REDACT = new Set(['password', 'token', 'authorization', 'cookie', 'secret'])

function safeFields(fields) {
  return Object.fromEntries(
    Object.entries(fields).map(([k, v]) =>
      REDACT.has(k.toLowerCase()) ? [k, '[REDACTED]'] : [k, v]
    )
  )
}

Ten lines in a shared logging package beats a thousand-line pipeline config. Keep ingest-time filtering as a second layer — a tripwire that alerts when something matching a credential pattern arrives, because that means the source-side control failed somewhere.

This is the same principle as masking in session replay: sensitive data should be stripped before it leaves the process that owns it, not cleaned up downstream after copies exist. The same logic applies to masking PII in session recordings — client-side before transmission, or it doesn’t count.

Log management security includes who can read the logs

Now the uncomfortable part. Even a perfectly redacted log system is a high-value target, because logs reveal architecture, internal hostnames, user identifiers, and business activity. And an imperfectly redacted one — which is every real one — holds credentials. So ask the questions you’d ask about database access:

• Who can read production logs? In most small companies: every engineer, forever, including the contractor whose engagement ended in March. Log platforms accrete broad grants because access is requested during incidents and never revoked after.
• Is access scoped? Can the frontend team read the payments service’s logs? Should they?
• Are the credentials to the log system itself strong? A shared Grafana admin login or an Elasticsearch endpoint with no auth on an internal network is a common finding. “Internal network” is not an access control.
• Are log backups protected like the logs? Snapshots and cold storage archives inherit the sensitivity but rarely inherit the controls.

My engineering opinion: treat log read access as production data access, because that’s what it is. Same approval flow, same periodic review, same offboarding checklist entry.

Who watches the log system’s own audit log?

The final recursion, and the one that separates a compliance checkbox from an actual control. If your log platform records who searched what — many can — where does that record go, and would anyone notice something strange in it?

This matters for two reasons. First, insider misuse: log search is a way to look up a specific user’s activity, tokens, or personal data without touching the production database, and it bypasses whatever auditing the database has. Second, attacker cleanup: an intruder with admin access to the log system can delete the evidence of their own access. That’s why write-once or forwarded-out audit trails matter — the log system’s audit log should be shipped somewhere the log system’s own admins can’t edit.

You don’t need an elaborate setup. Forward the audit stream to a separate bucket with a deny-delete policy, alert on audit-log gaps, and put a quarterly fifteen-minute review of “who accessed logs and is that grant still justified” on someone’s calendar. Small, boring, and it converts your log platform from an unwatched liability into something you’d be comfortable explaining in an incident review.

Because the incident review is where this all becomes concrete. “The attacker read our logs” is a survivable sentence. “The attacker read our logs, which contained two years of bearer tokens, and we have no record of what they searched” is a different meeting entirely.