retrofit
0.13.0
retrofit.sh

The oplog

What retrofit records: an oplog of schema changes, each change an op on one line, each element it names known by an identity that outlasts its name, and the schema replay builds from them. Principles has the rules retrofit upholds.

The premise

What retrofit records is how the schema changes. You record semantic operations (rename this column, cast this type, extract this wide table into two narrower ones), and retrofit writes the SQL from them.

This matters because some changes are identity-preserving, and DDL cannot show it. A column rename and a drop-then-add leave the same end-state schema. Only the change record says the column’s identity continued: the foreign key still points at it and its rows are still there.

The oplog

The oplog is an append-only file of operators. One per line. Four header lines, then the ops:

# retrofit-oplog-format: 1
# retrofit-model-version: 1
# engine-version: 17
# retrofit-oplog-form: merge
CreateSchema(01bd95b8116c, prev=000000000000, id=a9c670f51071, name="users")
CreateTable(b2c822ec011a, prev=01bd95b8116c, in=a9c670f51071, id=e295fa023ed9, name="people", reference_data=false)
AddColumn(45bcdb3fca16, prev=b2c822ec011a, in=e295fa023ed9, id=3befa1c90631, name="id", type="bigint", identity=by_default)

Long lines are wrapped here to make them easier to read.

One oplog file per registered schema. Plain text: git diffs it cleanly, you can read it in a code review, you can grep it.

Each line has four parts:

  • <opid>: the op’s id, 12 hex characters drawn at random when the op is appended. It names the act, not the content: running the same command twice appends two ops with two ids, and replaying the same commands on a fresh oplog draws new ones. An op keeps its id when it is reordered or rebased by reconcile. No two ops in a log share one; validate refuses a duplicate.
  • prev=<opid>: the op before it, so the ops form a chain through the file. The first op in a schema has prev=000000000000. An op’s id does not depend on its prev=, so revising an early op leaves the later ops’ ids as they are.
  • <role>=<value>: payload. Element references (in=, on=, columns=…) bind by ElementID, not by name. Strings, hex IDs, enums, lists.
  • id=<minted-id>: for an op that creates an element, the new element’s ElementID, or, on a reversal that brings a dropped element back, the dropped element’s (ElementID).

Reading an op line

Take this line:

AddColumn(45bcdb3fca16, prev=b2c822ec011a, in=e295fa023ed9, id=3befa1c90631, name="id", type="bigint", not_null=false, identity=by_default)
  • 45bcdb3fca16: the opid, drawn when the op was appended.
  • prev=b2c822ec011a: the op before it.
  • in=e295fa023ed9: the table this column is in, by its ElementID.
  • id=3befa1c90631: the new column’s ElementID.
  • name="id": the column’s name, which RenameColumn can change.
  • type="bigint": its SQL type.
  • not_null=false, identity=by_default: its attributes, as you wrote them. PostgreSQL’s rule that an identity column is NOT NULL is applied at replay, so the oplog keeps your input as written.

Replay reads this op, adds a column vertex under the table vertex and applies the identity-implies-not_null rule; snapshot emits the column declaration from the result.

ElementID

Every element an op creates has an ElementID: schemas, tables, columns, constraints, indexes, views and materialized views, types, functions, triggers, policies and extensions.

The operator that creates an element (AddColumn for a column, CreateTable for a table, …) gives it an ElementID, drawn at random.

An ElementID is permanent for the element’s lifetime. It survives renames. It survives MoveColumn. It survives ExtractTable and JoinTable. It is invalidated only by tombstone ops (DropTable, DropColumn, …), and it is not reused, with one exception: the reversal retrofit op revert appends brings a dropped element back under its ElementID when it returns as the same kind of element in the container it left. The op that re-creates it is marked same_element=reversal, and validate refuses an op that mints a known ElementID without that mark. Any other re-create mints a new ElementID.

This holds for containers too, not only leaves. A ExtractTable that leaves live columns unassigned (the survivor arm) does not consume its source: the source survives the transform in place under the same ElementID and the same birth op, holding the unassigned columns. Only the --into children mint fresh ElementIDs. --rest <name> renames that survivor in place, which is a name transition, not a new identity. The survivor’s PK constraint name, inbound foreign keys, triggers, privileges and PG table OID are all continuing facts about the same table, never re-created.

When you write op add RenameColumn on=users.email name=email_address, retrofit resolves users.email to the column’s ElementID and stores that ID in the rename op, not the name. Snapshot then renders the latest name for that ID. The foreign key that referenced the column referenced the ID, not the name; nothing about the FK has to change.

Ops bind to identity; a name is an attribute the latest op sets.

Find an element’s ID and history shows how to look an element up.

Snapshot vs. oplog

Two views of the same schema, and one store beside them:

  • Oplog: the append-only history. Authored. Versioned in git. Says how the schema came to be.
  • Snapshot: canonical DDL emitted by retrofit snapshot. Says what the schema is at any point in the oplog: the latest by default, an earlier one with --at. The same point always emits the same DDL.
  • Data files: the rows, in runnable .sql files kept beside the oplog (see Data files).

You commit the oplog. The snapshot is build output.

oplog (authored, versioned)
   │
   │  retrofit snapshot
   ▼
DDL (build output, deterministic, discardable)

Replay and genealogy

When retrofit needs to know “what does the schema look like right now?” (for snapshot, for op add validation, for drift detection), it replays the oplog through a deterministic engine that builds a genealogy forest: a content-addressed tree where each addressable element is a vertex, and each op contributes one or more edges.

Replay enforces structural rules (you can’t DropColumn a column that doesn’t exist, you can’t AddPrimaryKey to a column that’s already in another PK), normalizes SQL semantics (an identity= column is implicitly not_null), and produces vertex content hashes that anchor drift detection.

You don’t see the genealogy directly. You see its outputs: snapshot DDL, op add validation errors, drift questions.

snapshot renders whatever the replay builds. It keeps no list of the operators it supports: an op the replay accepts has a post-state, and snapshot writes it. A few ops have no DDL of their own:

  • Seal and DeclareReversal record facts about the log (its integrity, how a raw script is undone), not about the schema.
  • Drop* and Rename* change which elements the replay builds, or what they are called. Their effect shows in the snapshot as one element fewer or a new name, not as a statement of their own.
  • RunRawScript runs once, when a migration applies it. A DO block has no current-state DDL, so snapshot leaves it out and prints a note on stderr with the count it left out.

An element the replay builds can still fail to render; see Render failures.

What retrofit leaves to you

  • Applying the SQL. migrate and snapshot write their SQL to stdout and connect to no database; your pipeline applies it.
  • Keeping history. A Seal makes an edit to sealed ops detectable, and keeps no copy of them. The oplog’s history is in git.
  • Upgrading before 1.0. A release before 1.0 may change anything, and its release notes list each change; see Versions and compatibility.

Before 1.0, any release may change behavior; the release notes list what changed.

Published .