The oplog
What retrofit records: an oplog of schema changes, each change an op on one line, each element it names known by an identity that outlasts its name, and the schema replay builds from them. Principles has the rules retrofit upholds.
The premise
What retrofit records is how the schema changes. You record semantic operations (rename this column, cast this type, extract this wide table into two narrower ones), and retrofit writes the SQL from them.
This matters because some changes are identity-preserving, and DDL cannot show it. A column rename and a drop-then-add leave the same end-state schema. Only the change record says the column’s identity continued: the foreign key still points at it and its rows are still there.
The oplog
The oplog is an append-only file of operators. One per line. Four header lines, then the ops:
# retrofit-oplog-format: 1
# retrofit-model-version: 1
# engine-version: 17
# retrofit-oplog-form: merge
CreateSchema(01bd95b8116c, prev=000000000000, id=a9c670f51071, name="users")
CreateTable(b2c822ec011a, prev=01bd95b8116c, in=a9c670f51071, id=e295fa023ed9, name="people", reference_data=false)
AddColumn(45bcdb3fca16, prev=b2c822ec011a, in=e295fa023ed9, id=3befa1c90631, name="id", type="bigint", identity=by_default)
Long lines are wrapped here to make them easier to read.
One oplog file per registered schema. Plain text: git diffs it
cleanly, you can read it in a code review, you can grep it.
Each line has four parts:
<opid>: the op’s id, 12 hex characters drawn at random when the op is appended. It names the act, not the content: running the same command twice appends two ops with two ids, and replaying the same commands on a fresh oplog draws new ones. An op keeps its id when it is reordered or rebased byreconcile. No two ops in a log share one;validaterefuses a duplicate.prev=<opid>: the op before it, so the ops form a chain through the file. The first op in a schema hasprev=000000000000. An op’s id does not depend on itsprev=, so revising an early op leaves the later ops’ ids as they are.<role>=<value>: payload. Element references (in=,on=,columns=…) bind by ElementID, not by name. Strings, hex IDs, enums, lists.id=<minted-id>: for an op that creates an element, the new element’s ElementID, or, on a reversal that brings a dropped element back, the dropped element’s (ElementID).
Reading an op line
Take this line:
AddColumn(45bcdb3fca16, prev=b2c822ec011a, in=e295fa023ed9, id=3befa1c90631, name="id", type="bigint", not_null=false, identity=by_default)
45bcdb3fca16: the opid, drawn when the op was appended.prev=b2c822ec011a: the op before it.in=e295fa023ed9: the table this column is in, by its ElementID.id=3befa1c90631: the new column’s ElementID.name="id": the column’s name, whichRenameColumncan change.type="bigint": its SQL type.not_null=false,identity=by_default: its attributes, as you wrote them. PostgreSQL’s rule that an identity column is NOT NULL is applied at replay, so the oplog keeps your input as written.
Replay reads this op, adds a column vertex under the table vertex and
applies the identity-implies-not_null rule; snapshot emits the column
declaration from the result.
ElementID
Every element an op creates has an ElementID: schemas, tables, columns, constraints, indexes, views and materialized views, types, functions, triggers, policies and extensions.
The operator that creates an element (AddColumn for a column,
CreateTable for a table, …) gives it an ElementID, drawn at random.
An ElementID is permanent for the element’s lifetime. It survives renames.
It survives MoveColumn. It survives ExtractTable and
JoinTable. It is invalidated only by tombstone ops (DropTable,
DropColumn, …), and it is not reused, with one exception: the
reversal retrofit op revert appends
brings a dropped element back under its ElementID when it returns as
the same kind of element in the container it left. The op that
re-creates it is marked same_element=reversal, and validate refuses
an op that mints a known ElementID without that mark. Any other
re-create mints a new ElementID.
This holds for containers too, not only leaves. A ExtractTable
that leaves live columns unassigned (the survivor arm) does not consume its source: the source survives the
transform in place under the same ElementID and the same birth op,
holding the unassigned columns. Only the --into children mint fresh
ElementIDs. --rest <name> renames that survivor in place, which is a
name transition, not a new identity. The survivor’s PK constraint
name, inbound foreign keys, triggers, privileges and PG table OID are
all continuing facts about the same table, never re-created.
When you write op add RenameColumn on=users.email name=email_address,
retrofit resolves users.email to the column’s ElementID and stores
that ID in the rename op, not the name. Snapshot then renders the
latest name for that ID. The foreign key that referenced the
column referenced the ID, not the name; nothing about the FK has to
change.
Ops bind to identity; a name is an attribute the latest op sets.
Find an element’s ID and history shows how to look an element up.
Snapshot vs. oplog
Two views of the same schema, and one store beside them:
- Oplog: the append-only history. Authored. Versioned in git. Says how the schema came to be.
- Snapshot: canonical DDL emitted by
retrofit snapshot. Says what the schema is at any point in the oplog: the latest by default, an earlier one with--at. The same point always emits the same DDL. - Data files: the rows, in runnable
.sqlfiles kept beside the oplog (see Data files).
You commit the oplog. The snapshot is build output.
oplog (authored, versioned)
│
│ retrofit snapshot
▼
DDL (build output, deterministic, discardable)
Replay and genealogy
When retrofit needs to know “what does the schema look like right
now?” (for snapshot, for op add validation, for drift detection),
it replays the oplog through a deterministic engine that builds a
genealogy forest: a content-addressed tree where each addressable
element is a vertex, and each op contributes one or more edges.
Replay enforces structural rules (you can’t DropColumn a column
that doesn’t exist, you can’t AddPrimaryKey to a column that’s
already in another PK), normalizes SQL semantics (an identity=
column is implicitly not_null), and produces vertex content
hashes that anchor drift detection.
You don’t see the genealogy directly. You see its outputs: snapshot
DDL, op add validation errors, drift questions.
snapshot renders whatever the replay builds. It keeps no list of the operators it supports: an op the replay accepts has a post-state, and snapshot writes it. A few ops have no DDL of their own:
SealandDeclareReversalrecord facts about the log (its integrity, how a raw script is undone), not about the schema.Drop*andRename*change which elements the replay builds, or what they are called. Their effect shows in the snapshot as one element fewer or a new name, not as a statement of their own.RunRawScriptruns once, when a migration applies it. ADOblock has no current-state DDL, so snapshot leaves it out and prints a note on stderr with the count it left out.
An element the replay builds can still fail to render; see Render failures.
What retrofit leaves to you
- Applying the SQL.
migrateandsnapshotwrite their SQL to stdout and connect to no database; your pipeline applies it. - Keeping history. A Seal makes an edit to sealed ops detectable, and keeps no copy of them. The oplog’s history is in git.
- Upgrading before 1.0. A release before 1.0 may change anything, and its release notes list each change; see Versions and compatibility.