retrofit
0.13.0
retrofit.sh

retrofit drift

retrofit drift reads a .sql source that states a schema’s current shape, compares it with the schema the oplog records, and appends the ops (operators) that bring the oplog to that shape, asking a question where a change has more than one reading. It writes the oplog and the data files of the reference tables it captures, and never runs a statement against a database. Record a schema changed outside the oplog is the task guide.

Synopsis

From retrofit drift --help:

Usage:
  retrofit drift <source> [--schema <name>] [flags]

Description

drift replays the oplog to its last op and takes the schema that replay builds, the cut. It reads the source as a state, the schema as it is now, pairs each element of the source with one of the cut, and appends an op for each difference: Add* and Create* ops for what the source adds, Drop* ops for what it no longer has, and a rename, a cast or a data-motion operator where a pairing says one element became another. The source is usually a pg_dump --schema-only capture; any DDL that re-creates the shape will do. retrofit import reads a source as a list of changes instead, and records each as stated.

Sources and schema

A source is a path with a .sql extension, - for standard input, or a stream such as <(…) or a FIFO, and a run takes one. A stream that delivers zero bytes, as a pipe from a failed producer does, refuses before anything is parsed; a zero-byte .sql file is an empty source.

drift reads and writes one registered schema: the one --schema names, or the one the resolver chain selects. It registers none, and the oplog must already record the schema’s first shape; retrofit import records that.

Output

The run prints a plan header (drift plan: N additions, M removals, Q questions), then one item per change:

  • op-applied: a change with one reading, appended with no question.
  • question: a change whose shape is clear and whose intent is not (see Questions).
  • skipped: a statement drift appends no op for, with its kind, name and class. The class decides the exit status (see Skips and the exit status).

A summary line follows with the ops appended (or that would be), the questions, and the skips by kind and class. --show=additions hides the Drop* items and --show=removals the Add* and Create* items; the header, the summary and what is written do not change. Under --as=json the stream is a plan-header record, the item records, a drift-run-summary record with the tallies, and the terminal record; the Output contract gives each shape.

Questions

A change with more than one defensible pairing is a question, never a guess, and drift takes no --force. One table dropped and one added with another name and the same shape is appended as RenameTable when it is the only viable pairing; a tie is a question. Several drops and adds of overlapping shape, a column whose type and perhaps name changed, and a drop and an add that could be a rename are questions.

The question classes are table-rename, column-rename, column-cast, combined-rename-cast, index-rename and view-rename, and the structural ones copy-table, extract-table, join-table, merge-table, partition-table and departition-table (see Data-motion operators).

drift opens $EDITOR (or $VISUAL) on one answer line per question, the default answer already written, and appends the ops the saved answers name:

ActionAnswers
rename <element> <name>a rename question, with the new name
cast <element> [using="<expr>"]a cast question
accept <element> [<field>=<value>…]a structural question, with the operator drift recognized; the fields are fill-ins such as a join’s predicate and kind
drop-add <element>any question, with a drop and an add; each moved column’s ElementID ends and a new one is assigned

A combined rename and cast is two lines on one element, the rename first. Saving the buffer with no answer line cancels the run and writes nothing.

-i opens the editor even with no questions. --on-question sets what a question does: error refuses, editor opens the editor, and skip lists each question as a skipped item of kind unresolved-question:<class> and appends the ops drift could form without it. With no --on-question, a question opens the editor when $EDITOR or $VISUAL is set and refuses otherwise.

Dry run

--dry-run prints the report, opens no editor, and writes neither the oplog nor a data file. On a plan with no questions it forms and validates the ops as a real run does, so a refusal comes back with the real run’s code and exit, and exit 0 means the ops would apply. A plan with questions is checked against the default answers: only refusals no answer can change fail the dry run.

Drops

By default an element of the cut the source lacks appends a Drop* op. The op loses no row by itself; rows are lost when a script holding the DROP is applied.

--ignore-drops withholds every Drop* op, and --ignore-drops=<kinds> the named kinds: a comma-separated list of table, column, index, view, materialized-view, type, constraint, foreign-key, primary-key, not-null, identity and schema. Each withheld drop is still reported, as a deferred-drop item (! (deferred) in text), and the plan header counts them: M removals (K deferred).

An answer that would append a withheld drop refuses, all or nothing, naming each object: a drop-add on a structural question, an accept on a join that drops a source constraint or index, or an accept on an extract that consumes its source. A join’s surplus key column is the exception: --ignore-drops=column keeps it and the join is appended.

Skips and the exit status

Every skipped item has one of four classes:

  • decision: set aside for a stated reason, such as an object outside the schema, grants and default privileges, materialized views, COPY data, the pg_dump preamble, or a question skipped under --on-question=skip.
  • preserved: DML set aside, as import sets it aside.
  • gap: content no operator records, or that cannot be bound, including a skip kind no class lists.
  • internal-defect: a statement a parser version check set aside; the run still exits 0.

A run exits 0 over appended ops and decision, preserved and internal-defect skips, so exit 0 does not mean the source matched. A gap skip makes the run refuse with drift-unreconciled-skips, on a dry run and a real run alike, after the summary is printed; a real run’s ops, and a --seal Seal, are written first. --allow-capability-gaps accepts the gaps and exits 0. A pg_dump --clean dump needs it, because its DROP CONSTRAINT and DROP DEFAULT preamble statements skip as gaps.

A refusal or a parse error decides the exit ahead of any skip, and question and deferred-drop items never change it. An index, or ENABLE ROW LEVEL SECURITY, on a table drift pairs as a rename produces no item, so it never reaches the exit. The refusal’s hint names each gap kind; a view, policy or foreign key over a table this run’s answers settled is planned by the next run.

These skips name a constraint drift could not append, and what to do:

SkipWhenThen
add-fk-unresolved-refA foreign key’s referenced table cannot be bound: it is under an unanswered rename question, dropped in this run, missing the referenced column, or the key has a column-restricted ON DELETE SET NULL (cols) or SET DEFAULT (cols) action. The record’s reason names which.A later run appends it once the table is settled, or retrofit op add AddForeignKey does.
add-unique-unmodeled-facet, add-pk-unmodeled-facetA UNIQUE or PRIMARY KEY has INCLUDE or WITH (…), or a primary key is DEFERRABLE; or its USING INDEX index is undeclared or not a plain unique btree (the record’s reason and hint name it).Fix the index, or add the key with op add and its backing_index. Where an index an equal key already adopted covers other columns, drop the key and add it again.
add-check-unmodeled-facetA CHECK is declared NO INHERIT.Nothing records the flag.

Data-motion operators

CopyTable, ExtractTable, JoinTable and MergeTable are recognized from the difference between the cut and the source, each as a question:

  • CopyTable: the source table persists, beside a new table of the same shape.
  • ExtractTable: the source is dropped and two or more tables whose columns cover it are added, and the question offers accept. When the source survives and loses columns to one added table, the question offers only drop-add, and a drift-survivor-extract-unrecognized warning prints, where it can, the op add ExtractTable command that keeps the columns’ identities.
  • JoinTable: drops of disjoint columns and one add of their union, joined by a foreign key; accept takes the join’s predicate and kind.
  • MergeTable: dropped tables of one shape beside a live or added into. accept takes one predicate.<source> per source; leaving one empty answers the question with a DropTable per source.

A plain table that appears partitioned in the source, or the reverse, is a PartitionTable or DepartitionTable question. A table the source attaches to a parent appends AttachPartition, and one it declares free-standing appends DetachPartition. A change no partition operator records, such as a move to another parent or a changed bound, skips as pg-declarative-partition-convert-unsupported, with a hint naming the cause.

SplitTable is never recognized: its row predicates are in the migration script, not the schema. A split’s signature is a copy’s, so the copy-table question names op add SplitTable … --conditions= as the alternative.

Reference data

drift captures the rows of the tables declared as reference data at its new head, with no flag. retrofit schema ref-tables add <table> declares a table, and --ref <table> declares one in the run. A project that declares none captures nothing, and its data statements are listed as skips.

For each declared table, drift reads its INSERT statements, and reads a TRUNCATE or an unqualified DELETE as an empty observation; it does not capture UPDATE, DELETE ... WHERE or MERGE. A table whose rows differ from its chain is captured as the full observed table state, one data file per table, reported as captured <table> (N row(s)) with the rows added and removed; an unchanged one reads already-captured. Rows of a table not declared are never captured. Seed rows are not schema: retrofit data capture --label <label> <source> captures them.

A declared table drift cannot compare is reported with its cause, never read as a change:

ConditionReport
Its chain cannot be folded offline: an append whose body is UPDATE, DELETE or MERGE.not comparable: append at <op> ...
The source states no rows for it, as a schema-only dump does.reference rows not observed: N tables
Its declaration was removed (ref-tables remove).not compared: <t> (no longer reference data since op <op>), with the command that declares it again

A captured table that lost a key since its chain is still captured, with a reference-key-lost warning; one the source holds empty is captured empty, with a reference-table-emptied warning.

A capture where the table’s chain already has an unwitnessed version at the oplog’s head replaces that version: its files are rewritten in place, surplus pages deleted, and each reported overwrote @<op>. Only a commit keeps the earlier bytes. A witnessed version is never rewritten.

Captured files join the data corpus under the schema’s data_root; The data file gives the format. Row changes never change the exit status. A real run’s drift-run-summary record holds the per-table results in its capture member; a dry run prints one rehearsal record per table instead:

{"disposition":"captured","labels":[],"purpose":"ref","purposes":["ref"],"records":1,"rows":3,"supersedes_anchor":"7ca4c5df543b","table":"app.currencies","type":"rehearsal"}

Rows are read from INSERT statements, as pg_dump --inserts and pg_dump --column-inserts write them; a COPY ... FROM stdin block whose rows do not parse as SQL stops the run. Each table’s INSERT band in the source is bounded by the schema’s data_band_max_bytes (6291456 bytes by default), checked before the source is parsed. --table <pattern> narrows the tables the capture reads, and --skip-manifest <path> writes its skips as JSONL.

Seals

--seal appends a Seal over drift’s new ops after a successful run. Bare --seal takes its reason from the source (drifted to <basename>), and --seal=<text> gives one. A run that appends no op appends no Seal. The Seal passes the checks op seal runs on the range it covers before anything is written.

Objects the project does not own

drift removes every statement an [[exclude]] entry in retrofit.toml owns before it reads the source, so an exclusion alone never changes the plan or the exit status. Each entry that removed a statement is reported on every run, and a kept statement that depends on an excluded object refuses. Exclude objects the project does not own covers the entries.

Engine version

drift parses with the PostgreSQL 17 grammar, so PostgreSQL 18 syntax refuses as a parse error (PostgreSQL versions). It never changes the schema’s pinned engine version; a source whose ops need a later major refuses, and the hint names the retrofit schema engine command that moves the pin.

Options

From retrofit drift --help:

Flags:
      --allow-capability-gaps             accept skips that drop content retrofit has no operator for (exit 0 instead of exit 2; the drift-run-summary record is byte-identical either way)
      --allow-readers-moved stringArray   attest that the readers of the table or column with this qualified old name are handled by the time the rename applies (repeatable; byte-exact, never trimmed); every other rename still warns
      --data-root data_root               root that overrides the schema's data_root
      --database-at string                the cut the database these ops are written for is at, from which apply-risk is judged (else the latest Seal)
      --dry-run                           parse, diff, and emit the report; never open the editor; never write the oplog
      --extra-seed-path seed_paths        root that appends to the declared seed_paths
  -h, --help                              help for drift
      --help-all                          show extended help (vocabulary, semantics, caveats)
      --ignore-drops string[="*"]         suppress Drop op authoring; bare = all kinds, or CSV of: table, column, index, view, materialized-view, type, constraint, foreign-key, primary-key, not-null, identity, schema (suppressed drops still surface as deferred-drop events)
  -i, --interactive                       open the editor even when there are no questions
      --on-question string                question policy for unattended runs: error|editor|skip (default: editor when $EDITOR or $VISUAL is set, else refuse)
      --ref <table>                       declare <table> reference data (repeatable): a CreateTable this run mints is written reference_data=true; a table live at the run's starting cut gets SetReferenceData(value=true) appended after the run's DDL ops
      --schema <name>                     registered schema <name> (resolves via the chain when omitted)
      --seal --seal=<text>                append a Seal op covering drift's new tail (bare flag auto-derives reason from source basename; --seal=<text> overrides); refused as `op seal` refuses when the range holds an unwitnessed reference record
      --seed-path seed_paths              root that replaces the declared seed_paths
      --show string                       report-side filter on the live event stream: both|additions|removals (presentation-only; writes proceed for both directions) (default "both")
      --skip-manifest <path>              write the reference capture's skip manifest to <path> as JSONL
      --table <pattern>                   repeatable source <pattern> narrowing the reference capture (bare relname or glob)

retrofit data describes data_root. The global flags are in the Output contract.

Diagnostics

A refusal writes nothing, except drift-unreconciled-skips: a real run’s ops, and a --seal Seal, are written before it, and --allow-capability-gaps accepts the gaps that raise it (see Skips and the exit status). Refusal codes lists every code drift prints, with its exit status.

Advisory codes lists the warnings drift prints, with every other advisory.

See also

Record a schema changed outside the oplog, retrofit import, Exclude objects the project does not own, retrofit data, Operators, Output contract, The oplog.

Before 1.0, any release may change behavior; the release notes list what changed.

Published .