Log Aggregation with Loki

Configure Loki for structured log collection, 30-day retention, and trace correlation in ɳSelf.

ɳSelf ships Loki as part of the monitoring plugin bundle. Logs from every plugin service flow through Promtail into Loki, viewable in Grafana Explore.

What’s included

30-day log retention (auto-purged via compactor)
Structured JSON log format with level, trace_id, and plugin labels
Trace ID linking: clicking a trace_id in Loki Explore jumps to Tempo
Automatic LogQL alert rule for error rate spikes

Enabling the monitoring bundle

nself plugin install monitoring
nself build && nself start

Logs are visible in Grafana at http://localhost:3000 under Explore → Loki within 60 seconds.

Structured logging in plugins

All ɳSelf plugins use log/slog with JSON output. The standard fields are:

Field	Description
`ts`	Timestamp (RFC3339Nano)
`level`	Log level: `debug`, `info`, `warn`, `error`
`service`	Plugin name, e.g. `ai`, `mux`
`trace_id`	OpenTelemetry trace ID (auto-injected)
`plugin`	Same as service, used as Loki label
`caller`	File:line for quick navigation

Log level configuration

Set LOG_LEVEL in your .env.prod:

LOG_LEVEL=info   # default: info in prod, debug in dev
LOG_FORMAT=json  # always json in production

Querying logs in Grafana

Open Grafana → Explore → select Loki datasource.

# All errors from the AI plugin in the last hour
{nself_plugin="ai", nself_level="error"} | json

# Filter by trace ID (link from Tempo)
{job="containerlogs"} |= "trace_id=abc123"

# Error rate over 5 minutes
sum(rate({nself_level="error"}[5m])) by (nself_plugin)

Retention configuration

Default retention is 30 days. To change it, edit monitoring/loki/local-config.yaml:

limits_config:
  retention_period: 720h  # 30 days, change to e.g. 360h for 15 days

Then redeploy: nself build && nself start.

Alerts

The monitoring bundle ships a LogQL alert rule that fires when any plugin logs more than 10 errors per minute. Routed to Alertmanager → Slack (if configured).

See Runbooks for automated response.