Instance Lifecycle

How AngaraBase decides “who owns these data files”: instance identity, the Instance Lease, and the startup sequence. The lease is what gives you safe crash recovery and storage portability across hosts — moving an instance without a dump/restore.

Instance Identity

On initialization an instance gets a durable identity:

• cluster_id — UUID of the logical database cluster.
• instance_id — UUID of this specific instance.
• Data directory and transaction-log directory — the physical file locations.

Identity is stored in two places so it survives both a file fault and a catalog fault:

1. VERSION marker — a binary file with format version and IDs.
2. System catalog pages — reserved pages in base.adb with full metadata.

Instance Lease

The lease stops two instances from writing to the same files at once — a direct route to corruption. The lease record lives in SysCatalogMetaV0 inside base.adb pages and is updated atomically with full page images. That is why it works on NFS/SAN, where flock() is unreliable.

What the lease holds

Lease states

 [None]  ──acquire──▶  [Held]  ──heartbeat──▶  [Held]
   ▲                    │
   └──release/expired───┘
                        │ heartbeat stopped (crash, network partition)
                        ▼
                  [Expired]  ──takeover──▶  [Held by new instance]

• None → Held — first startup, or startup after a graceful shutdown.
• Held → Held — a periodic heartbeat (every 10s by default) pushes expires_at forward.
• Held → None — graceful shutdown releases the lease immediately.
• Held → Expired — heartbeat stopped (crash, network partition); the lease expires once the TTL elapses.
• Expired → Held — a new instance takes over after the TTL expires.

Startup sequence

1. Pre-flight — data directory exists and is initialized, VERSION marker is compatible, page size matches the binary.
2. Lease acquisition — load the system catalog from base.adb, then:
   • no lease → acquire immediately;
   • expired lease → take over (with a log warning);
   • active lease → fail-closed with an informative error (see below);
   • forced takeover → override an active lease (dangerous).
3. Recovery — replay WAL (file_bin backend), rebuild in-memory MVCC state, start the heartbeat.
4. Ready — start listeners (pgwire, admin), accept clients, keep the heartbeat going until shutdown.

Recovery modes

recovery_mode is visible in sys.identity and takes three values:

Shared storage (NFS/SAN)

The lease lets several hosts see the same files but lets only one write:

Host A ──┐
         ├── NFS/SAN ──▶ [data/] [txlog/] [base.adb with lease]
Host B ──┘

• Failover — Host B takes over if Host A dies.
• Maintenance/migration — move an instance between hosts without a dump.
• Testing — run safely against a copy of production data.

Limitations: a single writer at any moment; network partitions can cause a false lease expiration; network-storage latency affects throughput.

Lease controls (env variables)

The lease is configured by environment variables only — there are no dedicated config keys for it.

Guidance: on a network with unstable latency, raise TTL and heartbeat (e.g. 60/20); for fast dev iteration, lower them (15/5). Keep the heartbeat well below the TTL.

⚠️ ANGARABASE_FORCE_LEASE_TAKEOVER=1 is dangerous. It overrides an active lease, ignoring a live holder. If the previous instance is in fact alive, you get two writers on the same files and data corruption. Use it only when the holder is confirmed dead (process not running, host unreachable). Normally, just wait for the TTL to expire — takeover happens on its own.

Observability and diagnostics

-- Current lease holder and recovery mode
SELECT lease_holder_id, lease_holder_hostname, lease_expires_at, recovery_mode
FROM sys.identity;

-- Instance health
SELECT uptime_seconds, txn_commit_epoch_current
FROM sys.health;

Lease events are written to the server log: acquire, takeover, heartbeat failure, release.

Common issues

Security: the lease does not authenticate — filesystem permissions and network controls are still required, especially on shared storage.

Related

Concepts

• Storage engine — the datadir the lease protects.
• Transactions and MVCC — what happens to in-flight transactions on forced_takeover.

How-to

• Crash recovery — operational recovery procedures after a crash.
• Backup and restore — how to protect the datadir and take a snapshot.

Reference

• System views sys.* — sys.identity, sys.health for lease diagnostics.
• Known issues and SQLSTATE — the INSTANCE_* error section.