Logging

AngaraBase emits structured logs via tracing / tracing_subscriber (a key=value format) suitable for aggregation systems (ELK, Splunk, etc.).

Levels

Level	When	Examples
`error`	system failures, unrecoverable errors	data corruption, OOM, panic recovery
`warn`	recoverable failures, degradation, misconfiguration	failed heartbeat, audit-sink errors, fallback modes
`info`	operational events, state transitions	instance startup, lease acquisition, stats completion
`debug`	detailed diagnostics, internal state	micro-rescan progress, MVCC-recovery details, buffer pool stats
`trace`	very verbose, hot path	per-tuple processing, lock acquisition

Recommendations: info in production; debug while investigating an incident; trace for development only.

Setting the level

Verbosity and filtering are controlled by the RUST_LOG environment variable (read at startup, default info). The EnvFilter syntax is supported, including per-module filters:

# global level
RUST_LOG=debug ./angarabase-server --config-path /etc/angarabase/angarabase.conf

# per module: debug for the server, warn for TLS
export RUST_LOG="angarabase_server=debug,rustls=warn"
./angarabase-server --config-path /etc/angarabase/angarabase.conf

The filter is fixed at startup — to change the level, restart the server with a new RUST_LOG.

The logging.log_level setting (env ANGARABASE_LOG_LEVEL, config [logging]) is validated and visible in sys.settings, but the console subscriber listens to RUST_LOG. To actually change the server’s verbosity, set RUST_LOG and restart.

-- view the recorded setting value
SELECT name, value FROM sys.settings WHERE name = 'logging.log_level';

Log line format

2026-03-09T10:30:45.123Z INFO angarabase_server::stats stats_scheduler: auto_analyze_triggered table=users staleness_score=2.45

Timestamp — ISO 8601, millisecond precision.
Level — ERROR/WARN/INFO/DEBUG/TRACE.
Module — Rust module path (e.g. angarabase_server::stats).
Context — structured key=value pairs with an operation prefix (e.g. instance_lease:, micro_rescan:). Sensitive data is kept out of logs — IDs only.

Production

Rotation (logrotate)

# /etc/logrotate.d/angarabase
/var/log/angarabase/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 0644 angarabase angarabase
    postrotate
        systemctl reload angarabase-server
    endscript
}

Aggregation

ELK (logstash):

filter {
  if [program] == "angarabase-server" {
    grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{DATA:module} %{GREEDYDATA:context}" } }
    kv   { source => "context" field_split => " " value_split => "=" }
  }
}

Splunk:

[angarabase]
EXTRACT-level = (?<level>ERROR|WARN|INFO|DEBUG|TRACE)
EXTRACT-operation = (?<operation>\w+):

Useful greps

grep "ERROR" /var/log/angarabase/server.log | wc -l        # error rate
grep "instance_lease.*failed" /var/log/angarabase/server.log
grep "mvcc_recovery:" /var/log/angarabase/server.log

Performance impact

Level	Overhead
`error`/`warn`	minimal — always safe in production
`info`	low — the default level
`debug`	moderate — short-term, for investigation
`trace`	high — development only

AngaraBook