ScrivaDB | Blog

Embedding ScrivaDB in your Go program

Mon, 27 Jul 2026 00:00:00 GMT

Most databases make you run a server. You install a daemon, open a port, manage credentials, pool connections — and then talk to it over a socket, even when “it” is running on the same laptop as your program.

If your program is written in Go and is the only thing writing the data, that’s a lot of ceremony for no benefit. ScrivaDB’s storage engine is a plain Go library, so you can skip all of it and run the database in-process — no gRPC, no network, no separate daemon. Same durability, same query model, living directly inside your binary.

Two lines to a database

go get github.com/srjn45/scriva

import "github.com/srjn45/scriva"

db, _ := scriva.Open("./data")   // embedded durability defaults (fsync ~1s)
defer db.Close()

Open points at a directory. That directory is the database — the NDJSON segment files, the indexes, the metadata. There’s nothing else to stand up.

Keyed CRUD with your own keys

You bring the keys. ScrivaDB stores records under caller-supplied string keys, so you’re not forced to round-trip a generated id back into your own model:

sessions := db.MustCollection("sessions")

id, _, _ := sessions.InsertWithKey("sess-1", map[string]any{"status": "open"})
rec, _   := sessions.GetByKey("sess-1")

Collections are created on demand — MustCollection hands you one, materializing it if it’s new. No migration step, no CREATE TABLE.

Compare-and-swap, not locks

Concurrency is handled optimistically. Every record carries a revision, and you can make an update conditional on the revision you last read — a compare-and-swap. If someone else wrote in between, your update is rejected and you retry, instead of two writers silently clobbering each other:

rec, _ := sessions.GetByKey("sess-1")

// only succeeds if sess-1 is still at rec.Rev
_, err := sessions.UpdateIfRev("sess-1", rec.Rev, map[string]any{"status": "closed"})
if err != nil {
    // someone updated it first — re-read and decide
}

This is the right primitive for the embedded case: no lock to hold across a network, no deadlock to untangle — just “write only if the world hasn’t moved under me.”

The rest of the surface

Embedding isn’t a stripped-down mode. You get the real engine:

Secondary indexes for fast equality and range queries — the same query engine the server exposes.
Upsert, count, and exists for the common one-liners.
In-process Watch subscriptions, so your program can react to changes without polling (with a documented overflow contract so a slow consumer can’t wedge a writer).
A LoadJSONL bulk-import path for seeding.

The part that matters for your binary

Here’s the detail that makes embedding actually pleasant: the engine package pulls in no gRPC, no protobuf, no Prometheus, no cobra, no OpenTelemetry. A CI gate enforces it on every commit. Embedding ScrivaDB won’t quietly drag a server framework and a metrics stack into your dependency graph — you get a storage engine, and that’s it.

That’s the difference between “a database that also has a library mode” and a library that happens to be a database. The dependency discipline is a feature, and it’s tested like one.

When embedded is the right call

Reach for the embedded engine when:

Your program is written in Go and is the sole writer of the data.
You want durability and a real query model without operating a separate process.
You’re building a CLI, a desktop app, an edge/IoT daemon, or a service that wants local state without a database container beside it.

And when you don’t fit that shape — multiple processes need to write, or you want to talk to the data from another language — the very same repo builds a standalone server with gRPC + REST and client SDKs in ten languages. The storage engine is identical; only the front door changes.

warden — a supervisor for fleets of AI coding agents — embeds ScrivaDB exactly this way: its sessions, pipelines, and agent state live in an in-process store, with no database to run alongside the daemon. That’s the embedded case in one sentence.

Embedding guide — the full API, durability modes, and the Watch overflow contract.
Data model — keys, revisions, and CAS in depth.

Encrypting a database you can cat

Mon, 27 Jul 2026 00:00:00 GMT

ScrivaDB’s whole personality is that your data is just text. One JSON object per line, in a file you can cat, grep, and tar. That transparency is the feature.

Now add a requirement: sensitive fields must be unreadable on disk without a key. At first glance it reads like a contradiction — encryption is, definitionally, making bytes unreadable. So the design question isn’t “which cipher?” It’s: how do you encrypt the sensitive parts without breaking the transparency, durability, and query model that make the thing worth using?

The answer turned out to be less about cryptography and more about where you put it.

The threat you’re actually defending against

Be precise about the threat model, because it decides the whole design. ScrivaDB is an embedded, local-storage engine — the app calls Open(...) and reads and writes in the same process. There’s no separate untrusted caller to bypass. So the thing you’re defending is data at rest:

A stolen laptop or disk.
A leaked backup tarball or filesystem snapshot.
Another OS user cat-ing your data/ directory.

What’s explicitly out of scope: a compromised running process. While the DB is open, the key and the decrypted plaintext are in memory — at-rest encryption can’t help you there, and pretending otherwise just produces false confidence. Naming that boundary out loud is what keeps the design honest.

One boundary, and everything else stays dumb

Here’s the core move. Encrypt and decrypt happen at exactly one place: the collection API boundary — Insert, Update, Upsert, Get, and the scan path. The map of data that gets serialized to a segment file always holds ciphertext for encrypted fields. The plaintext exists only above that line, in your process.

The payoff is everything below that line never learns a key exists:

Subsystem	What it sees
Compaction	Copies opaque ciphertext blobs when it rewrites entries. Never needs the key.
Index rebuild	Reads ciphertext from segments — safe, because encrypted fields are never indexed.
CRC32C checksum	Computed over the stored (ciphertext) form. Still catches bit-rot.
Backup / snapshot	A tar of the segments is already ciphertext — encrypted for free.
Replication	Ships the stored ciphertext; a follower needs no key to replicate.

That table is the whole design. By choosing one choke point, encryption becomes a local concern instead of a cross-cutting one — no subsystem-by-subsystem retrofit, no key threaded through the compactor.

What a value looks like on disk

Each encrypted value is a self-describing string: a reserved marker, an envelope version, the id of the key that sealed it, and the payload.

<marker>:v1:<key-id>:<base64url( nonce || ciphertext+tag )>

The marker isn’t the human-readable enc:v1: you might guess — it’s a distinctive, rare magic string, and any user write that begins with it is rejected. That single rule turns “is this value encrypted?” into an infallible check on read, with no side-table and nothing to drift out of sync.

The cipher is XChaCha20-Poly1305. Two properties earn its place:

It’s an AEAD — confidentiality and tamper-detection in one primitive. A modified ciphertext is rejected, never quietly decrypted to garbage. That composes perfectly with ScrivaDB’s existing “detect corruption on read” stance.
Its 192-bit random nonce needs no counter bookkeeping. Append-only writes mean every insert or update is a fresh write with a fresh nonce, so reuse simply can’t happen.

Passphrase-derived keys go through Argon2id (64 MiB / 3 iterations), with a per-DB random salt stored — safely — in cleartext in meta.json. The salt reveals nothing without the passphrase.

The one real tradeoff

An encrypted field is opaque, so the engine can’t filter, range, sort, or index on it — and the design enforces that rather than letting you discover it as a silent bug. Try to build an index on an encrypted field and you get a typed error, not a sidx_*.json file quietly leaking the plaintext back onto disk. A filter that references an encrypted field is rejected at query-planning time, not left to never-match.

For the fields you’d actually encrypt — passwords, tokens, SSNs, secrets — this costs exactly nothing, because you never query by them. You look them up by some other key and read them back. Naming that as the one tradeoff, instead of hand-waving it, is what makes the feature trustworthy.

Reads don’t care about your policy

The detail I find most satisfying: reads are driven by the per-value marker, not by the current configuration.

A value that begins with the marker is decrypted (the key chosen by the id it carries). Anything else is passed through as plaintext.

Because every value self-describes, a collection in any mixed state — legacy plaintext from before you turned encryption on, new ciphertext, a half-migrated tail — reads correctly with zero coordination. The policy governs only what new writes encrypt; it never governs reads. That’s what makes migration a non-event: flip the policy, new writes conform immediately, and a background re-encrypting compaction pass backfills the old records on its own schedule. Reads are correct the entire time.

Key rotation falls out of the same mechanism for free — every blob names its own key, so old segments keep decrypting under the old key while new writes use the new one, and a compaction pass retires the old key when you’re ready.

The lesson

The interesting part of adding encryption to ScrivaDB wasn’t the cipher — modern AEAD is close to a solved problem. It was placing the boundary so that a human-readable, append-only, self-compacting store could gain confidential fields without any other part of it having to change. Encrypt at one line; keep everything below it key-oblivious; let each value say what it is. You end up with a database you can still cat — you just can’t read the parts that matter without the key.

What is ScrivaDB? — the transparency-first design.
Durability & backup — why an encrypted tar is a free encrypted backup.

How append-only survives a crash

Mon, 27 Jul 2026 00:00:00 GMT

Every storage engine looks correct until the power goes out mid-write. That’s the moment that separates “a file I write JSON to” from “a database.” So it’s worth asking of ScrivaDB directly: what actually happens if the process is killed halfway through a write?

The answer is deliberately unglamorous. There’s no journal replay, no multi-phase recovery, no fsck that walks the whole store. There are three simple rules that compose into a strong guarantee.

Rule 1: never modify a byte in place

ScrivaDB is append-only. An insert, an update, and a delete are all the same physical operation — a new line appended to the end of the current segment file. An update doesn’t seek back and overwrite the old record; it writes a new one with a higher revision. A delete writes a tombstone line.

This is the load-bearing decision for crash safety. If you never overwrite existing bytes, a crash cannot corrupt a record that was already on disk — there is no in-place mutation for it to interrupt. The worst a crash can do is damage the write that was in flight, and that write is always at the tail of the file, never in the middle.

seg_000001.ndjson
  {"id":"01J8...","op":"put","rev":1,"data":{...},"crc":"a1b2"}   ← safe, sealed by later writes
  {"id":"01J8...","op":"put","rev":2,"data":{...},"crc":"c3d4"}   ← safe
  {"id":"01J8...","op":"put","rev":3,"data":{...},"cr             ← crash hit here

Everything above that torn last line is untouched and intact. The blast radius of a crash is exactly one record: the one you hadn’t finished.

Rule 2: on open, truncate the torn tail

So when ScrivaDB opens a segment, it doesn’t trust that the file ends cleanly. It seeks backward from the end to find the last complete line — the last one terminated by a newline — and truncates anything after it. That half-written final record from the crash simply disappears, and the segment is left at a known-good boundary, ready to append to again.

This is why recovery is O(seek), not O(scan): the only place damage can live is the very end, so that’s the only place recovery has to look. A clean shutdown and a kill -9 converge to the same valid state on the next open — the crash just costs you the one unacknowledged write, which the caller never got a success for anyway.

Rule 3: checksum every record, verify on read

Truncation handles the write that was interrupted. But there’s a subtler failure: a record that was written fine and then rotted — a flipped bit from a bad disk, a dodgy cable, a cosmic ray. The bytes are all there and the line is newline-terminated, so truncation won’t catch it. Left alone, the engine would hand you silently-wrong data, which is worse than an error.

So every entry carries a CRC32C (Castagnoli) checksum, computed over its identifying fields and its data and stored on the line itself. On read, the checksum is recomputed and compared. A mismatch is surfaced as a corruption error — the read fails loudly instead of returning data that looks valid but isn’t.

CRC32C specifically because it’s cheap (hardware-accelerated on modern CPUs, so it’s nearly free on the read path) and it’s the right tool for the job it’s doing: catching accidental bit-rot, not defending against a malicious tamperer. (For that threat you’d reach for the AEAD tag that comes with encryption-at-rest — a different primitive for a different problem.)

Why the three compose

Put together, the guarantee is easy to state:

A crash costs you at most the last in-flight write — and nothing you received a success for. (Append-only + tail truncation.)
Silent corruption of a record that was already durable is caught, not returned. (Per-record CRC32C.)

None of these three rules is clever on its own. Truncating a partial line is obvious. CRC32C is decades old. Append-only is the oldest trick in the log-structured book. The engineering is in choosing all three together and letting a background compactor reclaim the space from superseded records later — so you get durable, crash-safe writes without a write-ahead log, and a store whose recovery path is short enough to reason about in a single sitting.

Boring is the goal. When the power comes back, you want your database to come up in a known state and get on with it — and to do that by truncating one line, not by replaying a journal you have to trust.

What is ScrivaDB? — the append-only model end to end.
Durability & backup — fsync policies and point-in-time recovery.

When a file is your database

Mon, 27 Jul 2026 00:00:00 GMT

Most projects start with the same small lie: “I’ll just write it to a JSON file for now.”

It works until it doesn’t. The file grows. Two goroutines write at once and you get half a record. A crash mid-write truncates the whole thing. You add a mutex, then a temp-file-and-rename, then a backup copy, then an index because a linear scan got slow — and somewhere along the way you’ve hand-rolled a bad database. So you give up and stand up PostgreSQL: a daemon, a schema, a connection pool, a migration tool, and a container in every environment — to store a few thousand records that fit in RAM.

ScrivaDB is the thing that belongs in the gap between those two. Durable and queryable like a database, but on disk it’s still just text you can read.

The problem it solves

The gap is real and most stacks ignore it. On one side, a flat file is honest and inspectable but has no concurrency safety, no crash durability, no query engine, and no integrity checks. On the other, a full RDBMS gives you all of that — and an operational tax you pay forever: a separate process to run and monitor, a wire protocol, credentials, backups, version upgrades.

ScrivaDB’s bet is that a large class of software — CLI tools, local services, IoT daemons, desktop apps, small web backends, embedded state stores — needs the guarantees of a database without the operational footprint of one. So it keeps the guarantees and throws the footprint away:

Append-only writes. Inserts, updates, and deletes are always new lines appended to a segment file. Nothing is ever modified in place, so a crash can’t corrupt an existing record — the worst case is a torn final line, which is detected and skipped on load. A background compactor later merges and deduplicates sealed segments.
End-to-end integrity. Every entry carries a CRC32C checksum. Silent bit-rot on disk is caught on read and surfaced as an error, instead of quietly handing you wrong data.
A real query engine. Secondary indexes give O(1) equality and range lookups; Find streams results with keyset pagination; aggregations run server-side. It’s not “load everything and filter in a loop.”
More than key-value. Caller-supplied keys, per-record revisions with compare-and-swap, upsert, TTL, optimistic transactions, and live Watch subscriptions.

How it simplifies local storage

The part that changes how it feels to operate is the on-disk format. A collection is a set of NDJSON segment files — one JSON object per line, appended in order. There is no binary blob, no page format, no WAL you can’t read. Your data is text:

$ scriva-cli insert users '{"name":"alice","age":30}' --api-key dev-key
  ✓ id=01J8...  key=01J8...  rev=1

$ tail -1 data/users/000001.ndjson      # your data is just text
  {"id":"01J8...","key":"01J8...","rev":1,"data":{"name":"alice","age":30},"crc":"a1b2"}

That one property collapses a surprising number of operational chores into tools you already know:

Inspect with cat, grep, jq, tail -f. No client, no REPL, no SELECT required to see what’s in there.
Back up by streaming the directory through gzip. Restore is tar xzf. Your backup is a file, not a pg_dump ritual.
Debug by reading. When something looks wrong, the source of truth is a text file you can open — not an opaque store you have to interrogate through the very layer you suspect.
Migrate and diff with line-oriented Unix tools and version control intuition, because every write is a line.

And it’s one small Go binary. No JVM, no Python runtime, no daemon zoo, no config file required to get started. scriva serve gives you gRPC on :5433 and REST on :8080 from a single process. Or skip the server entirely: go get the engine and run the whole database in-process, with no network and no separate daemon — keyed CRUD, CAS, secondary indexes, and in-process Watch, embedded directly in your Go program.

Where to use it

ScrivaDB is the right tool when:

You need persistence without operating a database. A CLI, a desktop app, an IoT or edge daemon, a small internal service — anything where “install and run Postgres” is heavier than the problem deserves.
Your working set fits on one machine. Single-node data that you’re happy to keep on local disk.
You value being able to read your own data. Auditability, debuggability, and “just look at the file” are features you’ll actually use.
You want one API from any language. REST and gRPC come from the same binary, and there are client SDKs across ten languages.
You’re writing Go and want it embedded. In-process storage with real indexes and Watch, and zero gRPC/Prometheus/OTel dependencies pulled into your binary.

warden — a fleet supervisor for AI coding agents — embeds ScrivaDB as exactly this: the local store for sessions, pipelines, and agent state, with no external database to run alongside it.

Where not to use it

Being honest about the edges is the whole point of choosing a small tool deliberately. Reach for something else when:

Your dataset is too large to compact on a single machine. ScrivaDB is single-node by design. If you need horizontal sharding, it’s the wrong shape.
You need multi-node consensus and automatic failover. Replication is leader→follower with manual failover — deliberately simple. There’s no Raft, no automatic leader election. That’s a feature for small deployments and a dealbreaker for high-availability ones.
You need complex relational queries. There are secondary indexes, ranges, and aggregations — but no joins and no SQL planner. If your access pattern is a five-table join, use a relational database.
You have very high-cardinality, write-heavy churn where compaction can’t keep up, or you need the maturity and ecosystem of a decades-old engine for a system of record at scale.

Picking a database is mostly picking your constraints. ScrivaDB’s constraints are “one machine, readable on disk, no daemon to babysit” — and in exchange you get durability, integrity, and a real query engine without the operational weight. If that trade matches your problem, it’s a very comfortable place to live.

Try it

What is ScrivaDB? — the two-minute tour.
Install the server and CLI.
Quickstart — from empty directory to first query.
Embedding guide — run it in-process from Go.