Skip to content

Replication & failover

ScrivaDB supports leader→follower replication for read scaling and standby redundancy. It is intentionally simple: asynchronous, with manual failover. Automatic election is out of scope.

Start a hot standby that replicates from a leader:

Terminal window
scriva serve --data ./follower-data \
--replicate-from leader.example.com:5433 \
--replicate-ca ./leader-ca.pem \
--replicate-server-name leader.example.com \
--replicate-api-key "$REPLICATION_KEY"

The follower:

  1. Bootstraps from a snapshot of the leader.
  2. Tails the leader’s committed writes — each tagged with a monotonic global LSN — and applies them through the normal write path, so its indexes end up byte-for-byte consistent.
  3. Resumes after a disconnect from its persisted applied-LSN, with no gaps or duplicates.

Replication is async. Lag is not automatically bounded: monitor ReplicationStatus and choose an operational staleness or promotion limit. A follower that falls beyond the leader’s retained replication buffer must re-bootstrap from a snapshot.

Cross-collection transactions replicate as a single transaction record (protocol v2) carrying every operation under one LSN, so a follower never sees a partial transaction. Followers built from this release advertise protocol v2 automatically. A leader will not downgrade a transaction to per-entry records: an older follower that does not speak v2 keeps receiving ordinary writes, but is refused with FAILED_PRECONDITION (REPLICATION_PROTOCOL_UNSUPPORTED) the moment a transaction must be shipped — upgrade the follower first.

The follower applies each transaction atomically and durably through its own cross-collection coordinator: readers see all of its operations or none, and the follower’s applied LSN advances only once the whole transaction is committed locally. After a crash or reconnect an already-applied transaction is recognised and skipped; a redelivery whose content differs from what was applied stops the follower with a divergence error (re-bootstrap it from a snapshot). Missing participant collections are created on the follower as legacy replication does. A promotion waits for an in-flight transaction apply, so a new leader never starts from a partially applied transaction. See design-xtx-replication.md.

A follower serves reads — Find, FindById, FindByKey, Aggregate — from its applied state, and refuses writes with a typed FAILED_PRECONDITION. So you can scale read traffic horizontally by adding followers.

Every server exposes ReplicationStatus (leader LSN, per-follower lag). Bound how stale a replica read may be by diffing a follower’s applied_lsn against the leader’s leader_lsn.

After a leader loss, promote a caught-up follower with the admin Promote RPC:

Terminal window
scriva-cli promote --api-key dev-key

Promotion:

  • Stops replicating and lifts the read-only guard, so the node accepts writes.
  • Is guarded by a lag check — a stale replica is refused (override with --force).
  • Is one-way. There is no automatic election; re-point the old leader as a follower of the new one if you want it back.

See Operations & security for TLS/mTLS, downgrade protection, and the operator runbook for the full failover procedure.