Logging in Production: Lessons, Patterns, and Code Snippets

Even the most brilliant application can become unmanageable without effective logging. In production environments, logs provide the narrative for understanding system health, user behaviour, and unexpected failures. This post walks through core concepts, common pitfalls, and practical code snippets to help you log like a pro in production‑ready applications.

Why Logging Matters

• Troubleshooting — Logs enable quick diagnosis of issues without reproducing them locally.
• Observability — Together with metrics and traces, logs give you a fuller picture of system behaviour.
• Auditability — Proper log retention policies help track compliance and trace events back to root causes.

Logs are your black box flight recorder — treat them as critical infrastructure.

Structured vs. Plain‑Text Logs

Unstructured, free‑form strings are hard to query or analyse. Instead, use structured logging — serialising data as JSON or key=value pairs. This makes it easy to parse, search and feed logs into aggregation systems like Elasticsearch, Datadog or Splunk.

Example (Node.js with Pino):

1// src/logger.ts
2import pino, { Logger } from 'pino';
3
4const logger: Logger = pino({
5 level: process.env.LOG_LEVEL || 'info',
6 formatters: {
7 level(label: string) {
8 return { level: label };
9 },
10 },
11});
12
13export default logger;

Structured logs let you reliably analyse events across clusters or microservices without brittle regex parsing.

Key Principles of Production Logging

1. Structure All Logs

Use JSON or key=value fields instead of unstructured strings so your log management tools can parse them.

2. Use Log Levels Intentionally

• error — Something failed.
• warn — Something unusual happened but execution continued.
• info — Standard runtime events.
• debug/trace — Verbose details; often disabled in production.

Avoid logging sensitive data at any level.

3. Correlation IDs and Context

Distributed systems can involve many services handling a single request. Generate a unique correlation ID at the edge (API gateway) and propagate it downstream so you can trace a request end‑to‑end.

Express.js middleware:

1import { randomUUID } from 'crypto';
2import { Request, Response, NextFunction } from 'express';
3
4export function correlationId(req: Request, res: Response, next: NextFunction): void {
5 const id = randomUUID();
6 req.headers['x-correlation-id'] = id;
7 res.setHeader('x-correlation-id', id);
8 res.locals.requestId = id;
9 next();
10}

Using the ID in handlers:

1// src/routes/example.ts
2import { Request, Response } from 'express';
3import logger from '../logger';
4
5export function exampleHandler(req: Request, res: Response): void {
6 const { requestId } = res.locals;
7 logger.info({ requestId }, 'Processing example request');
8 // … do work …
9 logger.info({ requestId }, 'Finished example request');
10 res.status(200).send('OK');
11}

4. Redact Sensitive Information

Logs must never expose secrets, tokens or personal data. Redaction should happen at the logger level:

1// src/logger.ts (extended)
2const logger: Logger = pino({
3 level: process.env.LOG_LEVEL || 'info',
4 redact: {
5 paths: ['req.headers.authorization', 'password'],
6 remove: true,
7 },
8});

5. Send Logs to a Central Aggregator

Local files are fine for development, but production demands centralisation. Aggregators like Elastic Stack (Elasticsearch + Logstash + Kibana), cloud logging (AWS CloudWatch, GCP Cloud Logging) or hosted services (Datadog, New Relic) offer full‑text search, dashboards and alerting.

6. Rotate & Retain Logs Strategically

Unbounded log growth can fill disks and increase costs. Use log rotation and retention settings appropriate to your compliance requirements.

Example logrotate configuration:

1/var/log/app/*.log {
2 rotate 7
3 daily
4 compress
5 missingok
6 notifempty
7 create 0640 appuser appgroup
8}

7. Handle Failures Gracefully

Never let logging failures crash your application. If a logger throws an error or the log server is unreachable, your app should continue running while alerting operators.

Logging Principles for Production

• Log with purpose: Every log line should have a clear reason to exist. Who needs this information? Will it help diagnose or understand system behaviour?
• Prefer structured logs: Free‑form text is easy to write but hard to search. JSON or key–value logs give you consistent fields for downstream tools.
• Level carefully: Use levels to control noise — trace/debug for verbose details, info for normal operations, warn for anomalies, error for failures, fatal for critical conditions.
• Context is king: Include identifiers such as user IDs, request IDs or correlation IDs to tie logs together across services.
• Guard sensitive data: Never log personal information, access tokens or credentials. Scrub or mask user‑provided input before logging.

Putting It All Together

Your production logging stack might look like this:

• Application — Generates structured logs with correlation IDs and log levels.
• Sidecar Agent — Forwards logs to an aggregator (e.g., Fluent Bit, Vector).
• Log Aggregator — Centralises logs for storage and analysis.
• Dashboards & Alerts — Visualise logs and notify on anomalies.

Combine logs with metrics and traces for deeper observability. The goal is to surface actionable insights without overwhelming your team with noise.

Logging Example: Complete Express Setup

1import express, { Request, Response } from 'express';
2import logger from './logger';
3import { correlationId } from './middleware/request-context';
4
5const app = express();
6
7// Attach correlation ID to each request
8app.use(correlationId);
9
10app.get('/health', (req: Request, res: Response): void => {
11 const { requestId } = res.locals;
12 logger.info({ requestId }, 'Health check');
13 res.status(200).send('OK');
14});
15
16app.listen(process.env.PORT || 3000, () => {
17 logger.info('Server started');
18});

Safe, structured, correlated logging with zero sensitive data.

Final Thoughts

Logging is not a side effect; it's part of your application's contract with operators and users. Invest in consistent, structured logging, clear log levels, context propagation, redaction of sensitive data, central aggregation and retention, and resilience to logging failures. When you get it right, logs become your superpower for maintaining stability, scaling confidently and communicating clearly with stakeholders.

A Note from Hana

At Hana, we rely on these logging strategies every day to keep our services reliable and transparent. Thoughtful logging enables us to deliver robust solutions, debug swiftly and stay responsive to our customers' needs. If you'd like to learn more about how Hana applies these patterns in production, keep an eye on our upcoming blog posts and community updates.

Happy logging!