retrofit
0.13.0
retrofit.sh

Principles

The rules retrofit upholds. They explain why the tool behaves the way it does, and why a few things you might expect to be configurable aren’t.

Each is enforced in code, tests or review.

For the mental model the principles act on, see The oplog. For a hands-on tour see Create a project.

1. Rows in the target database are the asset

The tool emits schema changes for live databases, whose rows have to survive the change. Five things follow.

  • “Data” means rows in the target database. Not bytes in the oplog. The oplog is the change recipe you’re authoring. Friction is reserved for the boundary where retrofit’s output reaches a database.
  • Validity is strict everywhere; risk is decided where it lands. Every way into the oplog (op add, op revise, import, a hand edit, a merge) refuses a log no valid log may hold, and a writer may also refuse a likely mistake it can name better than any later check. A valid but risky change is appended with its finding. A risk a boundary gates is refused at the boundary that decides what lands or runs: validate --fail-on-apply-risk in CI, a seal, snapshot or migrate. A consent there states a fact about where the output runs. A risk no boundary gates, such as a rename that breaks readers still using the old name, stays an advisory finding. A team may turn on an earlier stop at the writer; it only adds a refusal and never replaces the boundary’s, because an import, a hand edit or a merge does not pass through the writer.
  • The tool doesn’t apply DDL itself. snapshot emits canonical SQL to stdout; whether/how it reaches your database is your pipeline’s responsibility. Defaults at the emission boundary are non-destructive: --include-drop=none (the default) emits CREATE only; destructive prologues are explicit opt-ins via --include-drop=scoped or =schema.
  • Change operators preserve element identity. RenameColumn, CastColumn, MoveColumn, ExtractTable/JoinTable, CopyColumn all bind to the element’s ElementID, not its name. Foreign keys, indexes, and views referencing that ElementID don’t need updating across the change.
  • Lossy authoring surfaces a data-loss-on-apply advisory. Each shipped operator carries an information-preservation classification: additive, preserving, or lossy. When op add appends a lossy op (DropTable, DropColumn, DropView, CastColumn with an explicit using= clause, …) it emits a structured warning: data-loss-on-apply: … advisory naming the affected element. The op is still appended: the oplog is not the database, and the unsealed tail is freely revisable (principle #3). The advisory exists so you see what the apply will do before the apply happens.

2. Never silently ignore: fail or warn

If retrofit doesn’t understand something, you find out.

  • Unknown DDL surfaces as a structured skip-manifest entry, not a best-effort guess.
  • Unknown enum values error out at DDL-emit time, not silently drop the clause.
  • Unknown vocabulary (or a future model version) is named, not skipped: validate fails or ends unverifiable, and never passes the log as fully validated.
  • Ambiguous in-body column references error with candidates listed; ambiguous unresolved references in import route to the skip-manifest with a warning.
  • Unknown flag values fail before the verb runs.

If you see retrofit do something quietly that you didn’t ask for, it’s a bug.

3. Sealed = immutable; unsealed = freely revisable

The oplog has two regions and two completely different sets of rules.

  • Sealed range is locked. The seal pins a Merkle digest over the sealed ops’ canonical bytes. Editing a sealed op in place fails validate with seal-digest-mismatch. A seal is the team’s agreement that the range is stable, because branches and databases build on it; it records no deployment.
  • Unsealed tail is a developer scratchpad: op revise, op remove and editor-buffer line deletion are all by design. Friction here would fight the workflow with no asset to protect.
  • A seal detects edits; it does not prevent them. import --force re-bootstraps the whole oplog, sealed prefix included, with fresh opids, but only when --allow-sealed-rewrite=<seal opid> names each Seal it replaces, and the run reports each one. A hand edit inside a sealed range fails verification at the next validate. Witnessed reference data (retrofit data witness) takes the same posture: an edited sealed reference record fails the next validate with data-witness-mismatch, and snapshot and migrate refuse to emit with the same code.
  • Recovery from accidental oplog edits is git, not retrofit. The oplog is a text file in your repo, so your version control is the recovery surface. There are no backup files or undo logs.

4. Identity is minted once and durable

Every addressable thing in retrofit has an immutable ID.

  • Opids are minted when an op is constructed: a drawn event identity, not a function of the line. The same act performed twice is two ops with two ids, because it happened twice. An id, once minted, is never re-minted for an unchanged line: reorder it, rewrite its prev=, rebase it through reconcile, revise it back to a value it already carries, and it keeps the id it was born with.
  • ElementIDs are minted at element creation and stay fixed for the element’s lifetime (across renames, casts, extracts, moves). They are not reused, except by a reversal that brings a dropped element back, which marks its op same_element=reversal.
  • Param order is append-only. A new optional parameter is added at the end of an operator’s signature, so an op written before the catalog grew writes the same bytes after it. A release that changes a signature in any other way lists it under Log format in its release notes; see Versions and compatibility.
  • Genealogy vertices are content-addressed. The replay forest is reproducible from the oplog alone.
  • No producer may mint ops to shore up a partially defined operator. An operator whose outcome renders as different DDL under snapshot and migrate is still one op. The identities of the elements it derives are recorded on that op, and their shape is derived at render time. No producer appends sibling ops to complete it: a sibling can be hand-deleted, which would leave the contract half-applied. No runtime flag marks these operators; the rule binds whoever defines one.

Durable identity is what makes the change operators expressible.

5. Determinism end-to-end

One build, given the same oplog, writes the same bytes on every machine and every run. What may change between releases is on Versions and compatibility.

  • Canonical byte form for every operator. One build on two machines computes identical bytes for the same op.
  • Sorted iteration anywhere bytes leave the process. No map-iteration reaches stdout, the snapshot, or a digest.
  • Replay is total: given the same oplog prefix and target opid, every consumer materializes the same schema.

This is what makes the snapshot a build artifact you can throw away and recreate, and what makes digests trustworthy.

6. Atomicity at every write boundary

A failed command never leaves you with a half-written file.

Every oplog append, config write, and registry edit goes through an atomic write path: tempfile → fsync → rename (the atomic commit point) → fsync directory (best-effort). If the process is killed, the file on disk is either the old version or the new version. Never partial. The old-or-new guarantee rests on the atomic rename, not the directory fsync.

7. Machine-parseable output is non-negotiable

Every byte retrofit prints is meant to survive grep, jq, awk, and screen readers.

  • One output gateway. Direct prints from verbs are CI failures.
  • NDJSON under --as=json: zero-or-more {"type":"<name>",...} intermediate records followed by exactly one terminal {"type":"ok"|"failed",...} record. One discriminator field (type) names every record.
  • Errors land on stdout (not stderr) so one parser handles success and failure.
  • No color codes. No emoji. No --no-color flag. -q/--quiet and -v/--verbose are persistent root flags that control only human text advisories on stderr: -q suppresses the Courtesy tier, stacking it (-qq) drops the whole advisory tier, and verbose reveals the Info tier. This advisory axis is orthogonal to --as=text|json; --quiet never suppresses errors, results, or structured JSON, at any level.
  • Error codes are what a script switches on. A consumer matches the code, never the message text. A release that renames or removes a code says so in its release notes; see Versions and compatibility.

The full contract is in Output contract.

8. Comments and hand-edits are preserved

When it edits retrofit.toml or .gitattributes, files you also edit by hand, retrofit changes them line by line rather than re-serializing them, so your comments and formatting survive; only runs of trailing blank lines may normalize.

9. .gitattributes merge=union is a runtime invariant

Applies when the working tree is a git checkout. Outside a git tree, the check is skipped (no merge surface to protect). Concurrent appends from two branches need the line *.oplog merge=union text eol=lf in .gitattributes at merge time, not at write time. When retrofit detects an ancestor .git directory it checks for the line on every config load; the oplog-mutating and registry verbs (op add/op seal, schema create/schema remove) surface a gitattributes-missing warning when it’s absent; re-run retrofit init to restore the line.

The warning does not change the exit code.

Union applies wherever git merges locally, and on GitLab, whose merges honour the attribute. GitHub’s pull request merges ignore it, so two pull requests that each append to an oplog conflict there; the merging guide shows the resolution on the branch.

A VCS is never executed: retrofit relates to git only through the files it writes (.gitattributes) and passive detection of an ancestor .git directory. It never shells out to git or any other version-control binary, so no git install is needed.

10. Single-schema projects work without ceremony

The most common onboarding path (one project, one schema) should have no friction.

  • The first schema create auto-pins default_schema. Every schema-context verb then resolves to it without --schema.
  • Multi-schema projects resolve via each schema’s oplog root (the directory holding its <name>.oplog) or explicit [directory_scope] entries; the user types --schema only when the resolver chain can’t decide.
  • The six-step resolver chain (see concepts.md § The resolver chain) is the single contract. Every schema-context verb runs it; every verb names the resolved schema in its output (stderr in text mode, terminal envelope in JSON).
  • Ambiguity is never resolved silently. When the chain has no answer, the verb refuses with default-schema-unset. When CI needs to forbid silent fall-through, set RETROFIT_REQUIRE_EXPLICIT_SCHEMA=1.

11. Validation accumulates, then reports

Validators don’t stop at the first violation. A proposed op gets a single pass; all violations come back in one envelope. One iteration, one report.

Before 1.0, any release may change behavior; the release notes list what changed.

Published .