retrofit
0.13.0
retrofit.sh

Changelog

All notable changes to retrofit are recorded here, newest first, in the categories of Keep a Changelog. Before 1.0 retrofit promises no compatibility: a change is disclosed here, not prevented.

[Unreleased] records what the builds since the last release changed; a release renames it to its version. Its fixed Log format heading records every change to the oplog’s vocabulary, parameters, headers or seal bytes: what changed and what a user does, usually regenerating the log. CI fails a commit that changes internal/oplog/catalog/*.golden.json, internal/oplog/serialize/ outside its tests, or internal/seal/testdata/leafbytes_*.golden but adds no item under that heading itself (make logformat-check); with make hooks, the commit-msg hook refuses it at git commit. A commit whose serializer edit changes none of the vocabulary, parameters, headers or seal bytes may instead end its message with a Log-format: unchanged trailer; a reader refusing a log it used to accept is a format change. A change to either set of goldens always needs the item.

[Unreleased]

Log format

[0.13.0]

Added

  • op revert records a rollback in the log. After migrate --down has rolled a database back, it appends the reversal that down computes: the ghost’s ops after the log’s last op, the reference leg’s records, a declared reverse as one raw script, and the data-loss notes. A database migrated from the log then gets the same result, and a later down knows what was undone; migrate still never writes the log. Each appended op records reason="undoes <opid>", naming the window op it undoes, unless reason=<text> is given. Recording twice appends nothing, and recording after more ops land appends only the new reversal. The fork guard and the validate gate run before the write, and --dry-run previews what would be appended. A revert that is not at the log’s tail refuses as revert-not-at-tail, and one whose leading leg is off a Seal as revert-leading-leg-unsealed, each with a hint naming the way out. migrate --down reads what op revert recorded and advances its range past it, so a window is not undone twice.
  • A reversal that re-creates a dropped table, column or other element (the down’s ghost and op revert) keeps the element’s ElementID when the re-create passes the same-element predicate, so the element’s history continues where a fresh id orphaned it. The mint is marked same_element=reversal (see Log format), and --dry-run prints the mark. validate refuses an unmarked mint of an id the log already knows, and op revise refuses to edit a marked mint. op squash nets a drop and the mints that return its element as one group, and op seal reports a data loss a squash of its range would avoid.
  • A down over a DropTable or DropColumn says that the element it re-creates comes back without its rows, as a re-created-empty item of reverse-irreversible, beside the forward’s data-loss-on-apply line. A paired table’s stand-in names each column that comes back empty. A forward window that re-creates a dropped element says it comes back empty too.
  • schema database-at <cut> pins the cut the local database is at, in the self-ignored .retrofit/database-at; --clear removes the schema’s line, and bare it prints the pin in force and where it came from. op add, import and drift also take the pin as --database-at or RETROFIT_DATABASE_AT, and judge apply-risk from it (see Changed). A pin file that exists but cannot be read refuses as io-error rather than being rewritten.
  • op add RenameTable, op add RenameColumn, import, drift and op revert take --allow-readers-moved <old-name>, repeatable. It attests that the readers of the table or column with that qualified old name are handled by the time the rename applies, and withholds apply-risk-rename-in-use for that rename; every other rename still warns, and the warning’s hint names the flag. A value that answers no rename, including one a refused run never reached, is reported as having no effect.
  • op add CastColumn raises apply-risk-table-rewrite when the cast makes PostgreSQL rewrite the table, naming the dependents rebuilt with it, a partition’s own and a name-less index included.
  • op add and op revise refuse as body-resolve a name in a function, trigger, view or raw script body that neither the log nor the project’s declarations hold where the body binds it: a top-level script statement, an always-reached DO statement, a BEGIN ATOMIC body, a %TYPE or %ROWTYPE declaration, or a string-literal SQL body with no polymorphic argument. Where PostgreSQL binds the name only when the body runs (a PL/pgSQL statement, a polymorphic SQL body, a conditional branch), they warn body-name-absent instead. A name an operator took off the schema (MoveColumn, ExtractTable, a partition operator, DropTable and the like) is named in the refusal with that operator, and a held body citing it raises body-name-removed. import judges every body at the end of its source, and validate --history holds a range to the same check: a certain site fails, a conditional one is advisory. Objects the log cannot see, such as an extension’s functions, are declared in [[external]] entries in retrofit.toml.
  • op add warns schema-partition-tree-indexed for a CreateIndex on a partition root or middle, and for a PartitionTable of a table that already holds an index: identify skips an indexed partition tree, so it cannot place a database over one. Where that tree is the only difference left, the from-dump-no-match, from-dump-behind and --expect outcomes say so and that drift finds nothing else differing, and outside --expect say to name the cut with migrate --from. Outcomes, codes and exit statuses are unchanged.

Changed

  • op add, import and drift judge apply-risk over the window from the database’s starting state, as validate --fail-on-apply-risk does, rather than one op in isolation. The starting state is the schema database-at pin, else the latest Seal at or before the op, else root. A table created inside that window holds no rows when the database applies the op, so the empty-table codes (apply-risk-add-unique, apply-risk-add-not-null and the like) are withheld on it, and apply-risk-rename-in-use is withheld on an old name the starting state does not hold. A line saying a code was withheld on an assumed starting state prints once per run. A schema in a fork keeps the per-op verdict.
  • A repeated --schema on any verb refuses as flag-invalid-value, where the last value won: op seal --schema a --schema b sealed only b. --from, --to and --at still take the last value.
  • validate and every verb that folds the registered schemas (import, drift, snapshot, identify, element list, element history) refuse two [[schema]] entries rooted at one SQL schema as name-conflict, naming both. Identical imports had bound a cross-schema view on some runs and refused it on others.
  • identify refuses a log that breaks a replay rule with the rule’s own code and hint, as validate, migrate and snapshot do: a rule-no-live-references log gets the replace-or-drop-function recourse where it was refused render-blocked-on-partial-forest with no hint. A replay-classed entry this build cannot check refuses as unverifiable.
  • op add CreatePartition payment public p1 reads public as the schema, as CreateTable does, where it bound name=public and partition_by=p1. A partition named like the schema spells name=.
  • op revise of an op an unsealed Seal once covered says the op was sealed under that Seal, since unsealed, so a database placed there is a working copy. It said the op was never sealed and a database at the Seal was unaffected.
  • A RunRawScript body kept opaque by a cast to a managed type gets a hint for the cast’s shape: drop a 'literal'::app.kind cast where the literal meets a column of that type, or write one statement per arm of a CASE of string literals.
  • schema create of a name already taken refuses with a hint naming retrofit init --config <dir>/retrofit.toml for a second project, and init below a project it leaves alone notes that project on stderr.
  • validate no longer reports schema-stray-oplog for an oplog inside a nested project, a directory with its own retrofit.toml.
  • Wording: a scalar param spelled as a flag (--referenced_table t) refuses unknown-key with a hint spelling it referenced_table=<value>; the missing-required hint gives the param’s help; the SplitTable and MergeTable examples spell --condition=; the RunRawScript summary says when a body is bound; data scaffold --purpose says both values are experimental; and the run-once RunRawScript notes of snapshot and identify read in the singular for one op.

Fixed

  • op unseal no longer refuses a log for a verdict the log already held before the unseal, such as a sealed drop a newer replay rule refuses. It reopens the tail so the op can be repaired and the log re-sealed; the other verbs still refuse the log until then.
  • op unseal, op remove and op revise under --dry-run no longer say the write stands when the post-write sweep refuses: the hint says nothing was written.
  • A function whose argument, OUT, table or return type is a managed type or a relation’s row type, stored unqualified (f(mood)), is emitted qualified in CREATE, DROP, ALTER ... RENAME, COMMENT ON and the restored grants. snapshot and migrate rendered it bare, so the CREATE failed or bound a same-named type on the search_path. Built-in and unmanaged types stay as written, and identify resolves a dump’s qualified spelling against such a function.
  • drift no longer plans a DropFunction and a CreateFunction for an unchanged function whose signature pg_dump prints differently from the log: a managed type schema-qualified (SETOF, arrays, row types and OUT columns included), a long-form type (timestamp without time zone, int), or a typmod PostgreSQL discards (varchar(10), numeric(10,2)). New CreateFunction records store the folded spelling; sealed records keep their bytes and still pair.
  • op add refuses as rule-unique-name-in a CreateFunction PostgreSQL would read as an existing function: one that differs only by a discarded typmod (f(varchar(20)) beside f(varchar(10))) or by qualifying its own schema’s managed type (f(mood) beside f(public.mood)). Both were stored, and migrate then issued a second CREATE FUNCTION that failed.
  • drift and identify no longer plan a ReplaceFunction, or answer from-dump-no-match, for a LANGUAGE sql body the log rendered whose string literal sits under a CASE, an array subscript or an array_to_string separator. The CASE and subscript fix reaches CHECK, index predicate, view and policy bodies too.
  • identify places a database whose skipped run holds a function body that reads nothing (BEGIN RETURN 2; END) ahead of a later ChangePrimaryKey, CastColumn, AddEnumValue or RenameTable, and a body whose only mention of a renamed table is an EXECUTE string. It answered from-dump-no-match and routed to drift.
  • identify on a dump that matches no cut no longer runs the full placement search over a log holding RenameIndex, where one such op took nearly a minute.
  • identify no longer refuses the project’s own snapshot when [[exclude]] names a schema an extension is placed in, and no longer scans a large snapshot in quadratic time per candidate cut.
  • A log whose first op names a prev= the file does not hold no longer panics migrate, snapshot or identify: they replay it alone and judge its replay rules. validate and op seal still refuse it as structural-prev-unknown. With several registered schemas, snapshot, element list and migrate no longer refuse such a log as unverifiable with a hint naming a build that does not exist.
  • import --force that re-mints a schema’s ids warns for each other registered schema whose bodies still name the old ids, and validate then fails rule-exists-and-alive. snapshot refuses such a body as invalid-cut, naming its holder, where it wrote an <elem:...> placeholder into the SQL.
  • DropColumn of a column ExtractTable derived (a rejoin column, or a survivor’s foreign key column) refuses under its live dependents as on any table, where it was admitted and left a snapshot with a PRIMARY KEY on a missing column; drop the foreign key and primary key first. AttachPartition over such keys no longer fails snapshot with invalid-cut.
  • op revise refuses a changed RunRawScript body that op add would refuse (a bare statement, two statements, an empty body), in every form. It wrote them at exit 0, for migrate to emit outside the one-DO contract.
  • schema remove removes the schema’s block wherever it sits in retrofit.toml, in every spelling the loader accepts. It deleted the oplog at exit 0 but left the block when another [[schema]] followed. A removal that cannot take every registration refuses before the oplog is deleted.
  • schema remove and schema move honor an absolute oplog_root: remove deleted a path that did not exist and left the real oplog, and move refused as move-failed. schema move and schema engine rewrite the config line the loader reads, where a quoted or repeated key reported success and kept the old value.
  • The apply-risk-recorded-unsealed note advises retrofit validate --fail-on-apply-risk --from <cut>; the command it printed passed --schema, which validate does not take, and exited 2.
  • op info no longer says DropGeneratedExpression ends an element: the column and its values stay.

Log format

  • The 15 operators that create an element (CreateTable, AddColumn, AddPrimaryKey, AddForeignKey, AddUnique, AddExclude, AddCheck, CreateIndex, CreateView, CreateMaterializedView, CreatePolicy, CreateType, CreateFunction, CreateTrigger, CreateExtension) take one optional derived key, same_element=reversal, on a mint that returns an element a drop ended under its own ElementID. A line without the key is unchanged; no log needs regenerating. A build that predates the key refuses a line that has it, by name.
  • A value outside the declared set of a derived enum (same_element, and MoveColumn’s arm) now reads as an unknown key, so validate reports the line unverifiable and the verbs that render or write refuse it by key, where it was a kind-mismatch parse error. An authored enum still refuses.

Known issues

  • Ending a column that a later RunRawScript body names is refused only when the body is bound. For a body stored opaque, op remove or op revise of the column, or a merge whose other side removed it, writes the log, and validate accepts it. migrate then emits the script against a table without the column, so the migration fails whole at apply. validate --history fails a pull request that adds such a body, unless the name sits only in a conditional branch or the body does not parse; it only advises on a body already on the target that the pull request leaves naming a removed column. Apply the migration to a scratch database in CI (retrofit migrate --from root --to last, or from the latest Seal).
  • identify on a dump that matches no cut can take minutes on a log holding a MergeTable, JoinTable, UnjoinTable or partition operator, where a dump that matches answers in seconds.
  • drift asks the reversal question only when the database holds something logged after the drop. A database that has not yet run the log’s newest drop reads as intent, and the plan re-adds the item with no question. Review a drift plan’s re-adds against the log before accepting them.
  • In a project with several registered schemas, drift plans a DropForeignKey for an unchanged cross-schema foreign key that import recorded, skips the file’s key as add-foreign-key-cross-schema-unsupported, and exits 0. Review each DropForeignKey in such a plan with --dry-run before recording it.
  • op add, op seal and migrate read only their own schema’s log, so a drop in one registered schema is accepted, sealed and emitted while another registered schema depends on its target. validate fails such a drop when the dependent is a view, policy, function or other body in another schema, but accepts it when the dependent is a foreign key or a column typed by the dropped type. Apply the migration to a scratch database holding every schema before it lands.
  • migrate refuses as emit-render-failed a body that names an element of another registered schema, though snapshot renders it. The detail reads as if the element were dropped; it is live in the other schema’s log. No cut renders it. Use snapshot for a fresh database.
  • A trigger whose EXECUTE FUNCTION names a function in another registered schema imports with a function-synthesized-stub finding and renders the call without its schema, which fails at apply or binds a same-named function on the search path. Treat each such finding as a broken trigger.
  • In a project with several registered schemas, validate --fail-on-apply-risk --from <opid> resolves the opid in every schema, so the schemas that do not hold it refuse as opid-not-found. Name the cut with an anchor, such as a Seal, instead.
  • drift records a CreateFunction whose signature names a type it set aside, so the function fails at apply under --allow-capability-gaps. Record the type before running drift.
  • migrate --from root of a log imported from a pg_dump --schema-only file can create a function ahead of the table its string-literal LANGUAGE sql body or %TYPE declaration reads, which PostgreSQL refuses under the default check_function_bodies. snapshot of the same log applies. Use snapshot for a fresh database.
  • A released transition or cleanse data record, in a project that opts in to them, is protected only by its position in the log: an edit to one behind a Seal passes validate, and migrate emits the edited bytes. Nor is a record the release shipped told apart from a branch’s backfill integrated behind it, which a database already past that point never runs.
  • Before 1.0 the oplog may gain operators and header keys. A build refuses a log with a newer retrofit-oplog-format or a key it does not know, so CI and every developer upgrade together.

[0.12.0]

Added

  • drift over a database that skipped a logged drop, as an out-of-order deploy does, asks a reversal question where it re-added the item with no question. The plan line names the op it reverses and that op’s Seal. The answers are keep, which an unanswered question takes and which leaves the log as it is, and accept, which records the restore. A table, a column, a key, a constraint, a trigger and a rename back all take the question.
  • A RunRawScript DO body that names only elements its binder can follow is now recorded bound to them. A later rename renders through the body in migrate, op show, the down and reconcile’s diff, and op remove of an element the body names refuses as rule-exists-and-alive. A body the binder cannot read stays opaque.
  • The script migrate emits reads the privileges held on a table, view or column before a re-create drops it and restores them on the replacement, so a grant made outside the log stays. Columns moved by MoveColumn, ExtractTable, JoinTable or InlineTable are named as not restored.
  • drift and import without --seal print one apply-risk-recorded-unsealed note per run, naming the tables, columns and renames they recorded: apply-risk reads what the unsealed tail creates as new and empty, and a table recorded from a populated database is neither.
  • The docs gain a refusal codes page, generated from the binary’s code registry, with each code’s kind, its exit statuses and the commands that can print it.

Changed

  • migrate renders a ReplaceView the old way, one CREATE OR REPLACE VIEW, only when the view’s column names at the op’s input cut are a name prefix of the new list and every prefix column’s type is proven unchanged. Any other definition is a rebuild: it drops the view’s tracked readers and the view, creates the view with the new definition, restores its grants, and re-creates its triggers and readers, with a -- WARNING: line naming each reader that may fail and a -- post-deploy: REFRESH for each materialized view re-created empty. The rebuild takes ACCESS EXCLUSIVE on the view and each reader, and op add raises apply-risk-view-rebuild for it. A view whose prefix column reads another view’s or a materialized view’s output column now rebuilds even for a tail-column add, since the type cannot be proven. In the lendlog fixture the two ops b56bc83a6caf and 1ed10791196f render as rebuilds where they rendered one CREATE OR REPLACE VIEW before; drift and the down emit one ReplaceView, no longer a DropView and a CreateView, so the view keeps its ElementID.

  • validate --history gains the opaque-body check: it fails when the pull request adds a body retrofit stores opaque (a RunRawScript with no bound_params=, a string-literal or unresolved LANGUAGE sql function body, or a view, materialized view or expression slot whose text names no element), since a later rename or drop leaves what it names stale and only a database apply checks it. Each finding names the line, its operator, slot and cause, and the rewrite. A pipeline that keeps applying the range to a database before it lands adds --allow-database-apply: the check then lists its findings as advisory and does not fail. op add, import and op revise refuse nothing new; the cause finding they print for such a body names the check and the consent, and import now prints it for a string-literal function and a view whose query falls back.

  • reconcile no longer applies a forced order that the run did not ask for. It refuses as fork-forced-order, exit 1, naming the flag that asks for the order and what the order overrides, and the JSON error gains forced_orders. A bare reconcile reports each forced order.

  • migrate over a RenameColumn, RenameTable, RenameView or RenameMaterializedView that a plain LANGUAGE plpgsql function reads follows the ALTER with a CREATE OR REPLACE FUNCTION under the new name, since PostgreSQL never rewrites such a body. Where the same migration also renames a type in the function’s signature, it refuses and names the drop-before, create-after sequence.

  • drift no longer records a column whose type it set aside. A column typed by a CREATE TYPE the log lacks is a column-type-unminted skip, of class gap, in place of an AddColumn or CastColumn that failed at apply.

  • drift raises a merge question only between versioned or snapshot names, so a table prefix such as wp_ no longer makes every table one family. It asks an index rename only between equal definitions, and pairs rename questions the same way on every import of one file.

  • import and data capture print preamble: N statements before their summary or verdict, so the summary is the last line of stdout. The text, the JSON placement and the counts are unchanged.

  • validate --fail-on-apply-risk and migrate’s counsel count a table created in the window as populated once a MergeTable, a MoveColumn down, a splice-band data body or a staged load gives it rows. A range that passed may now exit non-zero.

  • spec_ref in op info and the SPEC line of op add <Op> --help print the Operators page URL, not a repository path. Help text names docs pages by title and says operator, not SMO.

Fixed

  • A rename now follows, and a drop now counts, the references in a function body that earlier builds left stale or did not see: quoted, three-part and qualifier names in LANGUAGE plpgsql, %type declarations, reads through a CTE or an aliased subquery, THEN results, IN lists, VALUES rows, row-SET sources and subscripts. RETURN QUERY and RETURN NEXT bodies and assignments to a parameter now bind. A read the binder cannot resolve makes the body opaque, where it had bound a shadowing variable or nothing.
  • A BEGIN ATOMIC body’s INSERT, UPDATE, DELETE and MERGE statements now bind, with each MERGE clause scoped as PostgreSQL scopes it. drift no longer proposes a ReplaceFunction for an unchanged one on every run.
  • A trigger’s UPDATE OF list and a bare WHEN or parameter DEFAULT now follow renames. The UPDATE OF columns are quoted, and drift orders a column drop after the triggers that list it.
  • CreateView and ReplaceView expand a * inside a CTE, a FROM subquery and every arm of a set operation, not only the top-level leftmost one. A rename that makes a derived relation’s read ambiguous now skips the view in snapshot, named, rather than writing one PostgreSQL rejects.
  • migrate --down now re-creates a dropped LANGUAGE plpgsql function and a dropped trigger under the target’s names, orders a re-created function at its own position, drops a window-created reader ahead of the column it reads, and renders a bound script’s bare relations under the declared search path or refuses.
  • A reference floor crossing its declaration’s removal now applies as a counted row delta when no key identifies it, writes a body=none floor as its empty state’s delta, deletes a dropped NULL key NULL-safely, and folds a partition child’s columns by its root’s ids. A keyless floor had duplicated its rows at every apply, at exit 0.
  • CopyTable and SplitTable clones of keys and indexes now render named. PostgreSQL had picked a numbered name when the derived one was held, and a later DropIndex or RenameIndex then hit the wrong object.
  • import keeps %TYPE on a function’s argument or return type and keeps a managed type named like a built-in’s internal name (public.bool). A function’s SET and RESET clauses fold as PostgreSQL’s proconfig does, and each element of its search_path list renders as its own literal. drift resolves SET search_path FROM CURRENT as import does.
  • migrate parenthesizes a column default the DEFAULT slot cannot read bare, such as now() AT TIME ZONE 'utc'.
  • import moves a drop past a later statement of the same file that clears the LANGUAGE plpgsql reader blocking it, as PostgreSQL applies it.
  • ops upgrade-sql-bodies keeps a function’s volatility, parallel safety, SECURITY DEFINER, SET, COST, STRICT, ROWS and LEAKPROOF, which it had reset to their defaults while reporting success. A body given to op revise now binds at the line’s cut, as one given to op add does.
  • drift orders a merge or join question after a surviving foreign key’s drop, names the operator on a structural question’s text line where it printed <?>, and reports an earlier op that replay refuses under that op’s own code.
  • validate --history no longer fails fate-kept on a revise of a revision the base holds or on a removal whose replacer stays on the chain. Its collapse-parent finding names the revert and a recovery that completes after a hand edit rewires a successor. validate reports its other findings beside a seal digest mismatch.
  • reconcile names a remedy for a displaced identity generation, and describes a partition bound as the DDL spells it and an owned sequence by the name its column renders.
  • identify on a dump matching no cut no longer searches every placement on a log holding a CopyTable, SplitTable or ExtractTable, or on a dump of an indexed partition tree. One such dump had run past 30 minutes.
  • op show notes the sealed ops its default scope hides, so a touching query over sealed history no longer reads as no history. data list marks an orphaned record as data check refuses it.
  • A refusal or hint no longer names a verb that refuses in turn, a flag the user did not pass, or a generic cause. op add and op revise list the accepted keys for an unknown key=value, import of only CREATE SCHEMA names schema create, a stream source prints as a bare - in recipes, a malformed foreign key target refuses under a registered code, and -v folds identical repair sites into one line with a count.
  • import --help says sources are never re-sorted and a glob expands lexically (V10 before V2), with --from-file for any other order.
  • drift on a 4,651-op dry run takes 31 s, where it took 3,619 s. validate on a log whose tables churn columns grows about 8x at 4x the operations, where it grew 16.2x. A three-file reconcile over a 2,311-op log peaks at 262 MB, where it peaked at 1.82 GB.

Log format

  • ReplaceView accepts any new column list: replay no longer refuses a narrowing or a reorder, and the view-replace-superset code leaves the catalog, the registry and the rule census. A sealed ReplaceView writes the same bytes and replays as before; the render now rebuilds the view where CREATE OR REPLACE VIEW cannot take the change. An older build refuses such a line as view-replace-superset; upgrade it to read the log.
  • A RunRawScript whose bound body names a quoted relation or column (app."Account", "isActive") is now bound, where earlier builds kept it opaque. Its bound sentinel writes each quote of the name doubled, and the line gains a new producer-only bound_quoted=[body] beside bound_params=, listing the body params that hold such a sentinel. An older build refuses the line by that unknown key, and its validate refuses it as body-bound-edited; upgrade it to read the log. A rename of such a name to one that needs quoting now renders it quoted instead of refusing body-render-unquotable. A line without a quoted name is unchanged, so a log needs no regeneration.
  • A bound body whose sentinel was hand-edited malformed now loads whether or not bound_quoted= lists it, and replay refuses it as body-bound-edited; earlier in this release it refused at load as body-bound-unmarked.
  • A plain LANGUAGE plpgsql function body now records the tables, views and columns it reads, as an ansi-sql body does, so snapshot writes it with a renamed element’s new name. A DropTable, DropView or DropColumn of an element such a body still reads refuses as rule-no-live-references, naming the function, at op add, where earlier releases accepted it. A log holding one is refused by that code wherever it is replayed to be checked or rendered (validate, snapshot, migrate, data check, data list and data witness among them); op show and schema engine still read it. data check still reads the sealed part of a log whose only such drop sits after its last Seal. The log’s bytes are unchanged. To read such a log, replace the function with a body that no longer reads the element (ReplaceFunction) or drop it (DropFunction) ahead of the drop, then regenerate the log or re-seal.
  • A trigger’s WHEN and a function parameter’s DEFAULT now record the managed functions they call, as PostgreSQL does. A DropFunction of a function either still calls refuses as rule-no-live-references, naming the trigger or the function, at op add, where earlier releases accepted it and the migration failed at apply. A log holding one is refused by that code wherever it is replayed to be checked or rendered. The log’s bytes are unchanged. To read such a log, drop the trigger (DropTrigger), or replace the calling function without the call (ReplaceFunction) or drop it (DropFunction), ahead of the drop, then regenerate the log or re-seal. A call to one of two overloads of a name, or an unqualified call whose name a built-in shares, still records nothing.
  • RunRawScript gains the optional, producer-written bound_params=[body, reverse_body] param, written after the reversal params and before reason. A build without it refuses a line that holds it as an unknown key; no log written today holds it, so nothing needs regenerating.
  • A body param whose text holds the bound-script form "<bound:…>" but that bound_params= does not list refuses at load as body-bound-unmarked, whatever the operator. A log written by retrofit never holds one. Restore the marker, or replace each "<bound:…>" with the name it stands for.
  • Replay refuses more drops PostgreSQL refuses, as rule-no-live-references, at op add and wherever a log holding one is replayed: a DropFunction while a partition key calls the function; a DropColumn or MoveColumn of a column a trigger’s UPDATE OF list names; a DropTable while a column is typed by its row type or a body names that type; and a drop of what a BEGIN ATOMIC body’s DML statement, a subscript or a %type declaration reads. The log’s bytes are unchanged. To read such a log, clear the reference ahead of the drop, then regenerate the log or re-seal.
  • A body that names a dropped element refuses as rule-exists-and-alive, for a view, materialized view, function, policy, trigger WHEN, column default, generation expression or partition key expression. Only a merge that orders the drop first reaches this, and reconcile --clean now refuses that order as fork-no-clean.
  • An index, or a PRIMARY KEY, UNIQUE or EXCLUDE constraint, named as another relation or index of its schema refuses as rule-unique-name-in. It failed at apply with relation ... already exists.
  • validate and op seal refuse a prev= cycle as structural-prev-cycle and a dangling prev= as structural-prev-unknown, exit 1, wherever in the file they stand. A merge resolution or a rebase can leave either.
  • A CopyTable or SplitTable that clones a key or index whose name cannot be derived refuses as copy-clone-name-collision, naming the workaround of dropping it first and re-adding it after. The EXCLUDE case had refused as rule-exists-and-alive.

Known issues

  • Ending a column that a later RunRawScript body names is refused only when the body is bound. For a body stored opaque, op remove or op revise of the column, or a merge whose other side removed it, writes the log, and validate accepts it. migrate then emits the script against a table without the column, so the migration fails whole at apply. validate --history fails a pull request that adds such a body. A pipeline that passes --allow-database-apply should apply the migration to a scratch database (retrofit migrate --from root --to last, or from the latest Seal).
  • identify on a dump that matches no cut can take minutes on a log holding a MergeTable, JoinTable, UnjoinTable or partition operator, where a dump that matches answers in seconds.
  • drift asks the reversal question only when the database holds something logged after the drop. A database that has not yet run the log’s newest drop reads as intent, and the plan re-adds the item with no question. Review a drift plan’s re-adds against the log before accepting them.
  • In a project with several registered schemas, drift plans a DropForeignKey for an unchanged cross-schema foreign key that import recorded, skips the file’s key as add-foreign-key-cross-schema-unsupported, and exits 0. Review each DropForeignKey in such a plan with --dry-run before recording it.
  • op add, validate, op seal and migrate read only their own schema’s log, so a drop in one registered schema is accepted, sealed and emitted while another registered schema depends on its target. Apply the migration to a scratch database holding every schema before it lands.
  • migrate refuses as emit-render-failed a body that names an element of another registered schema, though snapshot renders it. The detail reads as if the element were dropped; it is live in the other schema’s log. No cut renders it. Use snapshot for a fresh database.
  • A trigger whose EXECUTE FUNCTION names a function in another registered schema imports with a function-synthesized-stub finding and renders the call without its schema, which fails at apply or binds a same-named function on the search path. Treat each such finding as a broken trigger.
  • drift records a CreateFunction whose signature names a type it set aside, so the function fails at apply under --allow-capability-gaps. Record the type before running drift.
  • A released transition or cleanse data record, in a project that opts in to them, is protected only by its position in the log: an edit to one behind a Seal passes validate, and migrate emits the edited bytes. Nor is a record the release shipped told apart from a branch’s backfill integrated behind it, which a database already past that point never runs.
  • Before 1.0 the oplog may gain operators and header keys. A build refuses a log with a newer retrofit-oplog-format or a key it does not know, so CI and every developer upgrade together.

[0.11.0]

Added

  • op info --as json, for one operator or the --dump, gives each operator a verbs object: for op_add, import, drift, snapshot and migrate, whether that verb ships the operator. Each value is available, unavailable, or syntax-unavailable for an operator this build cannot spell.

Changed

  • A fresh text-mode snapshot loads only the rows a record states as its table’s whole content (a replace section, or a reference restate) between the tables and the constraints, as before, and fires no trigger on them. Every other record (an append or delta section, a seed restate) now runs in a second -- retrofit:snapshot-data v1 band after the whole schema, before COMMIT, with constraints, indexes and triggers live, as migrate --from root runs it, so its rows fire the enabled triggers and meet the foreign keys in authored order. Each band re-seats its own tables’ sequences. A note, snapshot-state-under-trigger, names each state table and trigger migrate would fire; a warning, data-statement-order-fk, names a foreign key the order meets (a state ahead of its parent’s statements, or a child’s statements ahead of its parent’s) with the recourse. --as=json, --data-only and the banner opid are unchanged.
  • migrate warns data-statement-order-fk for the same two orders in its trailing seed band, where every foreign key is live: a staged replace table whose key reaches a parent only the band’s later records supply, and a child’s records ahead of its parent’s. The warning names the table, its parent, the key and the recourse; a key declared INITIALLY DEFERRED raises neither, except under --envelope=independent. The script is unchanged.
  • The drift plan: header counts changes, between removals and questions: drift plan: N additions, M removals, C changes, Q questions. A change is any planned op the header does not count as an addition or a removal, such as a rename, a cast or a NOT NULL flip, so a plan made only of those no longer reads 0 additions, 0 removals. The JSON plan-header record gains a changes field.
  • transition and cleanse data records are experimental. Every verb refuses them unless retrofit.toml opts in ([experimental] purposes = ["transition", "cleanse"]), and warns on every use once it does: a released one is protected only by its position in the log. Write a backfill as a DO block to keep it under a Seal’s digest. Without the key, import --purpose and data scaffold --purpose refuse to mint either, and validate, data check, data list, data witness, data repin, migrate and snapshot refuse a corpus holding one, as experimental-not-enabled, naming each file and its purpose. With it, each verb reads and mints as before and warns experimental-purpose once per such file. The table is project-wide; an entry naming anything else refuses as config-invalid. A retrofit that predates the table refuses the whole file (unsupported table [experimental]).
  • migrate --down reports a row loss its render chose (data-loss-on-apply-render-chosen) as a warning, as data-loss-on-apply is, not as a note after the other warnings.
  • migrate --allow-unreached-dml, removed in 0.10.0, refuses flag-retired and says to drop the flag, instead of failing as an unknown flag.
  • op show --touching resolves a function, trigger, policy, extension or materialized view, by name, signature or element id, as element history does.
  • Refusals that pointed at generic advice now name their own recovery: a bare token in a payload record names the record’s fields (and ExtractTable’s --into); a body calling an overloaded function says no spelling binds it; a data file refused only for its record marker line leads with the marker grammar; reconcile --clean over a demotion whose other order replays clean names that order’s flags before --accept-dirty; schema engine <name> <version> names --schema.
  • A skipped sequence names itself on drift’s skip line and skipped event, and identify’s residual skip lines no longer end in a space.
  • after-traffic-outcome on a RunRawScript, whose declared reverse is opaque, says the down may refuse or may drop the rows written since, rather than that it does.
  • snapshot --help-all and docs/snapshot.md say that triggers and policies follow the data band, so a snapshot build runs no trigger on its seed rows while migrate --from root --label does.

Fixed

  • The drift plan: header counts the creation of a policy, function, trigger or extension as an addition and its drop as a removal, as it does for a table or a view; it had counted them as changes. --show=removals now hides those creations and --show=additions those drops.
  • migrate no longer emits a script PostgreSQL refuses with 55006 (pending trigger events) when an op writes a table holding a DEFERRABLE INITIALLY DEFERRED constraint trigger and the same transaction then alters that table, as AddNotNull … fill= does. Under --envelope own it refuses as migrate-deferred-trigger-pending, naming the trigger, the table and the ops; under caller the script and a deferred-trigger-owed warning name the SET CONSTRAINTS <schema>.<trigger> IMMEDIATE; the enclosing transaction can run first. A --down is treated the same way.
  • migrate now warns, as deferred-trigger-pending, where a data record writes a table holding a DEFERRABLE INITIALLY DEFERRED constraint trigger and a later op of the same transaction alters that table, which PostgreSQL refuses with 55006. The script has a -- retrofit: line above that op, and the warning names the SET CONSTRAINTS <schema>.<trigger> IMMEDIATE; the record can end with.
  • op seal, import --seal and drift --seal now refuse, as data-file-anchor-unresolved (exit 2), while the schema’s data corpus holds a record whose anchor-op= no longer names an op, as data check does, with the same detail and data repin’s recourse. Before, the seal was written, migrate refused the release it cut, and repairing the record needed op unseal first, since data repin refuses a sealed target. Regenerating a log under a committed data file, or op revise or op remove of the op a record anchors on, leaves such a record.
  • data check now refuses an oplog whose retrofit-oplog-format line is older or newer than this build reads, as oplog-format-too-old or oplog-format-too-new, exit 1. Before, it reported no findings and exited 0 on a log every other verb refuses, so a check run alone passed it. On a sealed log, validate now refuses the same line by the same code and the parser’s own hint, not sealed-oplog-unparsable with advice to repair the line.
  • op add PartitionTable, CreatePartition and AttachPartition now warn schema-partition-bound-timestamptz-zoneless when a range or list bound on a timestamptz key has no UTC offset (from=2024-01-01). PostgreSQL reads such a bound in the applying session’s time zone, so identify and drift cannot compare it with pg_dump’s output. The op still appends; the warning names the zoned spelling (2024-01-01 00:00:00+00). identify’s refusal detail now gives each skip’s hint under it, so a zoneless bound reads as needing an offset rather than as unsupported partitioning.
  • import --dry-run naming a schema that is not registered no longer creates it. The run still previews the import, then leaves the project as it found it: no <schema>.oplog, no [[schema]] row, no default_schema pin, and no new oplog_root directory. Previewing such an import no longer needs a retrofit schema create first.
  • snapshot --include-drop=scoped now drops the views and materialized views it writes, before the tables, so it re-applies to a database that already holds a schema with a view over one of its tables. Before, the script stopped at DROP TABLE because the view depended on the table, and rolled back. An object the prologue does not drop, such as a function or a view outside the snapshot, still fails the apply when it depends on a table or view the prologue drops.
  • import of DROP MATERIALIZED VIEW IF EXISTS naming a materialized view the import does not hold is now a counted no-op that exits 0, as DROP VIEW IF EXISTS is. Before, it was a capability gap and exited 2. IF EXISTS naming a table, view or index still exits 2, since PostgreSQL refuses it as not a materialized view.
  • snapshot --include-drop=scoped of a schema with a foreign key now applies to an empty database. The prologue’s foreign key drop reads ALTER TABLE IF EXISTS … DROP CONSTRAINT IF EXISTS …, so it does nothing when the table or its schema is absent. Before, it failed with schema "<name>" does not exist or relation "<name>" does not exist and the script rolled back.

Log format

  • A data file whose header says purpose=transition or purpose=cleanse refuses as experimental-not-enabled on every verb that reads it, unless retrofit.toml opts in with [experimental] purposes = ["transition", "cleanse"]. The file’s bytes and the oplog are unchanged; add the key to read such a log as 0.10.0 did, with a warning per file.

Known issues

  • Ending a column that a later RunRawScript body names is not flagged: op remove or op revise of the column, or a merge whose other side removed it, writes the log, and validate accepts it. migrate then emits the script against a table without the column, so the migration fails whole at apply. To catch it before landing, have CI apply the migration to a scratch database (retrofit migrate --from root --to last, or from the latest Seal).
  • identify on a dump that matches no cut can take minutes, where a dump that matches answers in seconds. In one measured case, a history of about 120 operations, a dump matching no cut took 137 s against 0.57 s for one that matched.
  • drift over a database that skipped a migration the log records as a drop, as an out-of-order deploy or a migration-folder tool running an older pending migration does, re-adds the dropped column as an AddColumn, with no question and no note. Review a drift plan’s re-adds against the log before accepting them.
  • A released transition or cleanse data record, in a project that opts in to them, is protected only by its position in the log: an edit to one behind a Seal passes validate, and migrate emits the edited bytes. Nor is a record the release shipped told apart from a branch’s backfill integrated behind it, which a database already past that point never runs.
  • Before 1.0 the oplog may gain operators and header keys. A build refuses a log with a newer retrofit-oplog-format or a key it does not know, so CI and every developer upgrade together.

[0.10.0]

Added

  • This change log. CI requires a “Log format” entry for any change to the oplog catalog golden, the serializer or the seal’s leaf-byte goldens. A serializer edit that leaves the format unchanged may say so with a Log-format: unchanged commit trailer instead.
  • Each release archive carries a man page per command and the bash, zsh and fish completion scripts, and the Homebrew cask installs them, so man retrofit-migrate works after brew install.
  • migrate and snapshot take --allow-unnamed-label <label> (repeatable). It attests that a labelled dataset was left unnamed on purpose, such as a test-only fixtures label on a long-lived target. The apply-risk-data-exclusions warning no longer counts that dataset, and its hint names this flag beside --label. Each value names one exact label, so a dataset added later still warns. An attestation that answers nothing prints a note that it had no effect.
  • CopyTable and SplitTable carry each EXCLUDE constraint of their source onto every table they create, under the name PostgreSQL gives it there (<table>_<keys>_excl); migrate adds it after the table’s LIKE. ExtractTable, JoinTable and UnjoinTable move an EXCLUDE under its own name onto the table that receives its key columns, and refuse one whose keys would land on two tables, naming it. MergeTable drops each source’s EXCLUDE with the source, and a merged row that breaks one on into fails the apply. A copy refuses an EXCLUDE whose name it cannot derive, or two that would take one name.
  • InlineTable carries an EXCLUDE constraint where it refused a table holding one. The survivor keeps its own, and a source’s moves onto the survivor under its own name, re-keyed onto the survivor’s key columns, as a UNIQUE does. migrate checks the rows the re-added constraint would hold by its own operators and refuses, naming it, when a padded row, or on the lookup arm two survivor rows sharing a folded row, would conflict, before anything is dropped. migrate --down of an ExtractTable over a table holding an EXCLUDE now inverts, re-adding it on the table it came from.

Log format

  • AddUnique takes an optional nulls_not_distinct=true for a UNIQUE NULLS NOT DISTINCT, which import now carries in every spelling where it skipped the key as unique-constraint, and drift where it skipped it as add-unique-unmodeled-facet. snapshot and migrate write UNIQUE NULLS NOT DISTINCT (…). drift recreates a key that differs from the target only by it (DropUnique, then AddUnique), and identify tells the two apart. Replay refuses the key beside backing_index, since USING INDEX takes the index’s NULLS treatment, and a keep over such a key refuses with reverse-policy-not-null: the NULLs it writes still collide. A UNIQUE that also carries INCLUDE, WITH (…), a tablespace or an access method still skips. A build without the key refuses a log that holds it, naming the key. A log that skipped such a key keeps the skip; re-import the statement to carry it.
  • The columns list of CreateView, ReplaceView, CreateMaterializedView and ReplaceMaterializedView percent-escapes each name’s %, , and ; (%25, %2C, %3B), so a quoted column name holding a comma, such as "revenue, usd", stays one column where it split into two. import, drift and migrate write the escaped form; op add’s columns= takes it too. A list whose names hold none of the three is unchanged, as is its content hash. A log that holds a literal %25, %2C or %3B in a view column name now reads it as %, , or ;; re-import that view. No other log needs regenerating.
  • CreateTrigger takes optional constraint=true, deferrable=true and initially_deferred=true for a CREATE CONSTRAINT TRIGGER, which import and drift now carry where they skipped it as trigger-unsupported. Its FROM form still skips as trigger-unsupported. Replay refuses a constraint trigger that is not AFTER, FOR EACH ROW, on a table and without transition tables, initially_deferred without deferrable, either one on a plain trigger, and RenameTrigger of a constraint trigger. Its name is also one of its table’s constraint names: a CHECK or key of that name refuses with rule-unique-name-in, and an unnamed one takes the next free derived name. import records ALTER TRIGGER … RENAME of one as the gap constraint-trigger-rename-unsupported, and refuses a trigger created over one’s name, or one created over a plain trigger’s (constraint-trigger-replace-invalid). snapshot and migrate write CREATE CONSTRAINT TRIGGER with its DEFERRABLE clause, and migrate counts it among the constraints a deferred key’s name must not share. A build without the keys refuses a log that holds them, naming the key. No log needs regenerating.
  • CreateFunction takes an optional procedure=true for a CREATE PROCEDURE, which import and drift now carry where they skipped it as create-procedure-unsupported. A procedure shares the functions’ names and signatures; replay refuses returns, volatility, parallel_safety, strict, cost, rows or leakproof on one, and a trigger that executes one. snapshot and migrate write CREATE OR REPLACE PROCEDURE, and name it as a PROCEDURE in DROP, ALTER … RENAME and COMMENT ON, each argument type marked IN. import reads the PROCEDURE and ROUTINE forms of DROP, ALTER and COMMENT ON; with the FUNCTION form, a name written without an argument list names the one routine of that name and kind, of any arity. It refuses what PostgreSQL refuses: an unmarked argument list two procedures answer to, or a name without a list that more than one routine answers to (routine-name-not-unique), and a statement naming a routine of the other kind, a CREATE OR REPLACE across kinds or a trigger on a procedure (routine-kind-mismatch). A build without the key refuses a log that holds it, naming the key. No log needs regenerating.
  • CreateFunction and ReplaceFunction take optional rows= (a function’s ROWS n, kept as written, like cost=) and leakproof=. import and drift carry both from CREATE FUNCTION, where they skipped the function as function-rows-unsupported or function-leakproof-unsupported, and import folds an ALTER FUNCTION … ROWS or … LEAKPROOF into a ReplaceFunction; drift still skips a target ALTER FUNCTION, as it does … COST. A build without them refuses a log that holds them, naming the key. No log needs regenerating.
  • A log holding one of the PostgreSQL 18 surfaces (AddColumn generated_kind=virtual, enforced=false, period=true, without_overlaps=true, SetConstraintEnforced, SetConstraintInherit) now fails validate with syntax-not-available, and no verb renders it. Regenerate the log without them, or revise each op to its PostgreSQL 17 form (generated_kind=stored, enforced=true, period=false, without_overlaps=false).
  • RunRawScript takes reverse_noop=<reason> and reverse_blocked=<reason> beside reverse_body, at most one per line, and a new DeclareReversal line (target=, supersedes=, one of those arms) declares a sealed script’s reversal without changing a sealed byte. A build without them refuses a log that holds them, naming the key or operator. No log needs regenerating.
  • reconcile --base --ours --theirs reads an input that has no engine-version header, and merges the key like any other: an absent key is a value. Every other command still refuses such a log, and no log written before needs regenerating.
  • reconcile --base --ours --theirs keeps a side’s own followers’ prev= bytes when the other side removes the line they follow: the removed line stays in that side’s place as its tombstone. No log needs regenerating.
  • The three-file mode’s refusal hints name the inputs by role (ours, theirs) and its own re-run, never a flag it refuses.
  • reconcile --base --ours --theirs refuses fork-write-write for a value both sides change that no block order decides (one side revising a base op in place, the other setting the same property); --keep or --swap writes the stated outcome. No log needs regenerating.
  • The conflict-marker and duplicate-opid refusals name the three-file mode (reconcile --base --ours --theirs --result) with its inputs by role, and a partition-scoped column facet set against its parent’s recursing op refuses fork-write-write naming <partition>.<column>. No log needs regenerating.
  • reconcile --base --ours --theirs writes --swap’s order with a block’s tombstones beside the block they follow, so --swap over (base, ours, theirs) writes what --keep over (base, theirs, ours) writes, and reads a line a side holds as the base does through the other side’s revision of its predecessor as unchanged. No log needs regenerating.
  • AddIdentity, DropIdentity and SetIdentityGeneration take an optional partition=, a partition beneath the column’s owner whose own identity the op adds, drops or changes in place on a PostgreSQL 15 or 16 pin. op add writes it for a <partition>.<column> token there; from 17 on the op refuses on a partition’s column. No log needs regenerating.
  • AddColumn and AddIdentity take optional identity_start, identity_increment, identity_minvalue, identity_maxvalue, identity_cache and identity_cycle=true for an identity’s sequence options, which import now carries on an inline identity column, ADD COLUMN and ADD GENERATED where it skipped them as identity-options-dropped. An option that restates PostgreSQL’s default for the column’s type and direction is dropped; AS, OWNED BY, RESTART and a partition’s own identity still skip. Replay refuses an option without an identity (identity-option-without-identity), one that is not an integer (identity-option-not-integer), and a bound outside the column’s type, MINVALUE not below MAXVALUE, a START outside them, INCREMENT 0 or CACHE below 1 (identity-option-out-of-range). snapshot and migrate write the options in the identity’s sequence list; drift carries the target’s options onto an identity it adopts and reports identity-options-differ for a paired one whose options differ. A build without the keys refuses a log that holds them, naming the key. A log that skipped such options keeps the skip; re-import the statement to carry them.
  • SetIdentityOptions changes an identity column’s sequence options (ALTER COLUMN … SET START WITH | INCREMENT BY | [NO] MINVALUE | [NO] MAXVALUE | CACHE | [NO] CYCLE). It takes identity_start, identity_increment, identity_minvalue, identity_maxvalue, identity_cache, identity_no_minvalue, identity_no_maxvalue and identity_cycle (false is NO CYCLE); an option it does not name keeps its value. import writes it for those forms, which it skipped as AlterTableSetIdentity (RESTART still skips), and drift for an identity whose options differ from the target’s, which it skipped as identity-options-differ. A build without the operator refuses a log that holds it. No log needs regenerating.
  • SetIdentityOptions takes an optional partition=, as SetIdentityGeneration does, and a partition-scoped AddIdentity takes the identity_* options it refused: on a PostgreSQL 15 or 16 pin a partition’s own identity keeps its sequence options through ATTACH and DETACH, which dropped them. import writes them for an ADD GENERATED or ALTER COLUMN … SET on an attached partition, which it skipped as identity-options-dropped or AlterTableSetIdentity, and drift plans the scoped SetIdentityOptions for a partition whose own identity’s options differ from the target’s. A log that skipped such a statement keeps the skip; re-import it to carry the options.
  • AddExclude and DropExclude add and drop an EXCLUDE constraint. AddExclude takes method (btree, gist, spgist or hash), the key list as CreateIndex spells it (columns, key_expressions, key_opclasses, column_directions, nulls_ordering), key_operators (each key’s WITH operator, && or OPERATOR(schema.op)), name, deferrable and initially. import writes them for ALTER TABLE … ADD CONSTRAINT … EXCLUDE, the table-level EXCLUDE inside CREATE TABLE and DROP CONSTRAINT of one, where it skipped the first as alter-table-other and the second as inline-exclude-constraint. An EXCLUDE with a WHERE, INCLUDE, WITH (…), USING INDEX TABLESPACE or a key COLLATE, or on a partitioned table or a partition, still skips, now as exclude-constraint in both spellings. snapshot and migrate write it, and AddExclude raises apply-risk-add-exclude. InlineTable, PartitionTable and DepartitionTable refuse a table that holds one, and AddExclude refuses a partitioned table. A build without the operators refuses a log that holds them. A log that skipped an EXCLUDE keeps the skip; re-import the statement to carry it.
  • drift reads an EXCLUDE in both spellings and pairs it with the log’s by its keys, method, operators, operator classes, sort order and deferral, not by name. A difference in any of them plans DropExclude then AddExclude, and a name-only difference plans RenameConstraint. An operator pg_dump qualifies by the search path’s first schema (OPERATOR(public.===)) compares equal to the bare ===. A target EXCLUDE with a facet AddExclude cannot carry is an exclude-constraint skip and drops nothing. identify keys the same facets, so two cuts differing only by an EXCLUDE’s operator are not one match. Before, drift reported every EXCLUDE the log held as an exclude-constraint skip and ignored the target’s.
  • AddExclude takes optional include_columns, storage_params and predicate for an EXCLUDE’s INCLUDE (…), WITH (…) and WHERE (…), each spelled as on CreateIndex. import and drift now carry the three where they skipped the constraint as exclude-constraint, and snapshot and migrate write them. An unnamed EXCLUDE with INCLUDE takes PostgreSQL’s name over its keys and its INCLUDE columns (t_x_n_excl), where import held its name over the keys alone. A WHERE that does not resolve skips the whole constraint, since without it the constraint compares every row; a WITH (…) value the log cannot spell, USING INDEX TABLESPACE and a key COLLATE still skip. drift plans a change to any of the three as DropExclude then AddExclude, and DROP COLUMN of a column only the INCLUDE or WHERE names drops the constraint with it, as PostgreSQL does. A build without the parameters refuses a log that uses them. A log that skipped such an EXCLUDE keeps the skip; re-import the statement to carry it.
  • PartitionTable takes two optional lists, partition_parents and partition_keys, one entry per partition, to declare a tree of several levels; a single-level record is unchanged and carries neither. Nothing to regenerate: an existing log replays as it did. op show --framing=add writes a partition bound or key as one JSON object, which op add takes back.
  • Every oplog names its form on a fourth header line, # retrofit-oplog-form: merge or # retrofit-oplog-form: compact. A log without the line is refused (oplog-form-header-missing), as is any other value (unknown-oplog-form). In the merge form, which schema create writes unless --form compact is given, a removed or revised operation stays in the file as a ~ tombstone or behind a replaces= list; op compact drops those traces. Every read checks each live Seal over its chain, so a Seal whose digest no longer matches is refused. For an existing log, add # retrofit-oplog-form: compact as line 4, after # engine-version:, then run retrofit schema form merge to keep the history from then on.
  • op show’s runnable framings print CreateIndex’s key_opclasses as --key_opclass, not --key_opclasse. The log’s bytes are unchanged; re-run op show for any copied --key_opclasse line.
  • A # retrofit-* header key this build does not know is refused on every verb that loads the oplog (unknown-key, writes-blocked-on-unknowns or render-blocked-on-unknowns by the verb’s role), naming the key and its line, instead of being carried through opaquely. Run a build that knows the header; do not delete it.
  • ExtractTable declares the rule extract-partitioned-source: an extract that moves every live column refuses a partitioned parent or a partition as its source. A log holding such an extract no longer replays. Regenerate the log, running DepartitionTable or DetachPartition before the extract, or leave a parent’s partition key on it so the source survives.
  • AttachPartition ends the partition’s own columns into its parent’s, and DetachPartition mints the detached table’s columns and CHECKs afresh, recording their IDs in its new optional column-ids and adjacent-ids. SetDefault, DropDefault, AddNotNull, DropNotNull and SetColumnComment take an optional partition (partition=<table>, or the on-column spelled <partition>.<column>) that scopes the op to one partition of the column’s family. New refusals: partition-not-beneath-owner, partition-not-null-inherited, partition-identity-absent, detach-id-list-arity and attach-partition-check-mismatch. A log whose ops name an attached partition’s own columns, or a detached table’s columns by their earlier IDs, no longer replays: regenerate any log that holds an AttachPartition.

Fixed

  • migrate --down over a window that drops a table holding an EXCLUDE constraint, or merges such a table into another, re-creates the table with each EXCLUDE under its name. It refused the window migrate-non-reversible where the same window over a UNIQUE inverted. An EXCLUDE a CopyTable or SplitTable cloned still refuses. The down of an ExtractTable that inverts as an InlineTable still refuses a table that holds an EXCLUDE, and now names the constraint.
  • reconcile <fork-selector> --clean on a fork nested in another fork’s block closes that fork alone. It also hung the enclosing fork’s other block onto the result, closing the enclosing fork in an order nothing checked, and when that block’s lines came first it moved the nested fork’s ops under it. The enclosing fork now stays a fork, which the next reconcile --clean writes or refuses fork-no-clean. The fork-nested refusal and the bare report’s next step name the nested fork’s selector to close first, where they said no reconcile invocation could resolve it.
  • op add ChangePrimaryKey requires name= for the new key, as AddPrimaryKey does, unless it sets backing_index, and drift and import give a rotated-in key PostgreSQL’s <table>_pkey. An unnamed key was added bare, so the name PostgreSQL gave it depended on where an earlier migrate window ended, and a later window could drop or rename a name the database did not hold. A log that already holds an unnamed ChangePrimaryKey still reads.
  • migrate reports a seed record no --label named as not-selected even when it is anchored behind --from. It used to print a dml-behind-window warning for it, and --allow-unnamed-label noted that it had no effect, so a fixtures dataset kept warning on every later window. A selected record behind the window still warns dml-behind-window.
  • The dml-chain-near-miss warning a data mint raises compares a new chain only with existing chains of its own band (reference, seed or history). data capture --label fixtures on a table that also holds a cleanse chain no longer warns that a label drift forks the chain, or suggests re-running with the other chain’s label-set: a chain of another band is never the one a capture supersedes.
  • data capture’s verdict line prints not comparable: once before the reason instead of twice, and data capture --dry-run rehearses a table it would capture as would capture:, not would captured:.
  • migrate’s data-replace-prologue and data-replace-cascade warnings say a replaced table “is replaced by its full observed state” and replacing <table>, not “cleared” and “clearing”: a keyed replace upserts the observed rows and deletes only the keys the state lacks, and never empties the table.
  • op add CreateIndex --help spells the per-key operator-class flag --key_opclass, which op add and op revise accept beside --key_opclasses. The misspelling --key_opclasse is refused.
  • init against an existing .gitattributes prints a note naming each line it appends, instead of changing the file silently.
  • A data file’s record marker line that leaves out a required token (kind=, seq= or the src= provenance) is refused naming the missing token, not as carrying an “unrecognized or mis-ordered” one. Every marker-line refusal (G-e) now spells the marker grammar as its remedy, and the shared repair hint names the data file; data check and the other corpus readers no longer append a line about a “sidecar”.
  • identify and migrate --from-dump refuse from-dump-schema-absent for a dump of a database that holds neither the (non-public) schema nor any of its objects, and point at migrate --from root. They matched such a dump to the CreateSchema cut, so migrate --from-dump emitted a script without CREATE SCHEMA that failed when applied.
  • identify and migrate --from-dump list the schema facets that did not pair (cut only: and dump only: lines in the detail) when the dump matches no cut but drift would plan nothing against the nearest one. The from-dump-no-match hint no longer sends the user to a drift --dry-run that reports 0 ops, and the from-dump-behind hint no longer says such a divergence was made outside the log; both offer the nearest cut by hand with migrate --from.
  • retrofit completion <shell> prints the shell’s completion script. The root’s help listed it, but running it was refused as an unknown command, and the scripts’ completion requests were refused the same way.
  • Every verb that loads the oplog refuses an operator this build does not know, a ~ tombstone included, naming the operator and the line, with the hint “upgrade the CLI to one that knows the operator; do not delete or edit the line”. The op verbs and validate refuse it as unknown-operator, except that validate keeps sealed-oplog-unparsable for one inside a sealed range. A renamed or split operator keeps its re-import or rename remedy.
  • validate reports an unknown key once, naming its line, the key and the rules it leaves unverifiable, instead of one unverifiable row per rule for every op of the operator. No hint says to hand-fix the op. It also reports a header key this build does not know, and a retrofit-model-version newer than the build, as unverifiable (exit 2); an unsealed log with a newer model-version no longer passes.
  • reconcile exits 2, not 1, for a file only a newer build can read: an unknown operator, a too-new retrofit-oplog-format or retrofit-model-version, an unknown header, and (as before) an unknown key. The JSON code tells them apart; a conflict marker, a repeated header or a refused fork stays exit 1.
  • reconcile, the op verbs and op revise -i refuse an oplog that exists but cannot be read (for example mode 000) with oplog-unreadable (exit 2), printing cannot read oplog and the read error, instead of oplog-not-found (“no oplog”) or a bare command-error.
  • validate refuses an oplog holding an op that needs a newer engine version than its engine-version pin, with unsupported-on-engine-version (exit 2), as migrate and snapshot do. A log whose pin was lowered by hand, as in a resolved header merge conflict, no longer passes the CI gate.
  • op seal and identify refuse the same log with the same unsupported-on-engine-version (exit 2): op seal writes no Seal over it and identify resolves no cut. op info drops the note that said they did not check the engine-version gate.
  • validate, migrate and the other commands that read the log refuse a CreatePartition carrying both partition_columns and partition_expression, as they already refused it on CreateTable and PartitionTable. Such a sub-partition used to replay, and migrate rendered its key from the expression alone, dropping the columns. A CreateTable carrying both with no partition_by is refused too, rather than read as a plain table with its key dropped.
  • op add and the commands that read the log refuse an ExtractTable that moves every live column of a partitioned parent or a partition, with extract-partitioned-source. Such an extract ended the source while a parent’s partitions stayed live, or while a partition’s parent still named it, and the rendered DROP diverged the database from the log.

Changed

  • Every own-envelope migrate file now carries an open-transaction check: a head ahead of BEGIN; and a recheck after the ON_ERROR_STOP guard. This enforces the own contract, that the file owns its transaction. Applied inside a transaction that is already open (psql --single-transaction, a runner’s own transaction, psql AUTOCOMMIT off, a single-Sync batch), the file refuses by name, own-envelope-transaction-open (SQLSTATE 25001), before any DDL. That includes cells that happened to work before, such as a lone unsegmented file under psql --single-transaction, and cells that applied but lost the file’s atomicity at exit 0. The check needs USAGE on plpgsql. Every own file’s bytes change, so a checksum recorded over an earlier own file (Flyway’s, Liquibase’s) no longer matches a re-emit.
  • PostgreSQL 18 is a supported engine version, like 15, 16 and 17, at the PostgreSQL 17 syntax level: every verb writes and reads the SQL it does at 17. The PostgreSQL 18 syntax refuses syntax-not-available (exit 2) at every engine version: AddColumn generated_kind=virtual, enforced=false on AddForeignKey and AddCheck, period=true on AddForeignKey, without_overlaps=true on AddPrimaryKey, AddUnique and ChangePrimaryKey, and the SetConstraintEnforced and SetConstraintInherit operators. op add and op revise refuse it, and validate, op seal, identify, snapshot and migrate refuse a log that holds it, and drift refuses to append to one. op info marks these values and operators as not available.
  • migrate refuses a forward window whose reference record is superseded by a later replace of its table while a step between them reads the table (dml-superseded-in-window, exit 2). Before, such a window exited 0 and the dependent step ran on rows the pruned record would have filled, so an AddNotNull could fail at apply. data capture and drift warn with the same code when the floor they mint makes such a window, the snapshot capture hint names the split and the version, and op seal refuses a range that would seal the pair.
  • migrate renders a PartitionTable of several levels as one nested PARTITION OF tree behind a LOCK TABLE ... ACCESS EXCLUSIVE and a guard that refuses, before any change, rows the declared tree does not route or that a CHECK of the window rejects. A DepartitionTable rebuild opens with the same lock, and the target of one must be the root: a leaf or middle refuses departition-target-is-partition. Its warnings now walk every descendant, naming each partition-owned index, constraint, policy and trigger (an inherited copy is not named), each partition’s stamped DEFAULT, NOT NULL and comment, and each kept serial sequence the rebuild drops; they say the down across it refuses for now. The generated-column recompute warning says the same. ExtractTable of a column that a partition-owned element or a middle’s partition key names refuses, as it did for the source’s own.
  • op info shows, for each rule, what retrofit does to an op that breaks it at two moments: OP ADD (op add and the other commands that write an op) and READING THE LOG (migrate, snapshot, validate, op seal, identify), each refuses or lets through. --as json keeps when and adds enforcement: {"append": ..., "replay": ...}, derived from it.
  • op info says READING THE LOG refuses for partition_cols_xor_expr on CreateTable, CreatePartition and PartitionTable, and for view-replace-superset on ReplaceView: the commands that read the log refuse an op that breaks them, which op info said they let through. --as json reports these rules with when: append+replay, enforcement.replay: refuses and binding: binding, no longer author-time-convenience.
  • MergeTable’s rules list MergeShape on sources: the refusal it already raised, when a source’s columns differ from into’s or a source has a nextval default or no live column, now has a declaration.
  • Advisory messages end without a period. The second sentence of data-motion-partition-not-pg-declarative and of move-column-down-refused, the advice, moved to the hint. A migrate --down script’s reverse-unrecorded line ends ; the database sits at <opid>. The five witness warnings on a record the down reverts state their cause instead of repeating their code, and data-motion-source-consumed, dml-body-partition-dependent, data-only-single-statement-autocommit and the two drift-skip warnings are reworded in plain terms.
  • data-loss-on-apply messages end without a period too: the reasons op add, migrate and migrate --down build for a drop, a cast, a join, an unjoin, a matview replace and a lookup extract or inline. Drift’s extract-recognition diagnostics read “an ExtractTable”.
  • The Advisory codes page shows every form of a message that varies with a field, each led by the value that selects it, and follows a message that is only a placeholder with its detail and hint.

Removed

  • migrate --allow-unreached-dml and its dml-ahead-of-window warning. A data record anchored after --to is outside the window and is now omitted without a refusal, a warning or a flag, as a reference or seed record past --to already was. Promoting an environment that sits behind the tail no longer needs the flag; drop it from scripts, where it is now an unknown flag. dml-anchor-unresolved now means only an anchor the oplog does not hold, fixed with retrofit data repin.
  • LayerSeparation and NotSealed no longer appear in any operator’s rules: retrofit never checked either. Layer separation is a property of the oplog’s slots (docs/concepts.md, “The oplog”); the sealed region is guarded by sealed-region-write.

Known issues

  • Ending a column that a later RunRawScript body names is not flagged: op remove or op revise of the column, or a merge whose other side removed it, writes the log, and validate accepts it. migrate then emits the script against a table without the column, so the migration fails whole at apply. To catch it before landing, have CI apply the migration to a scratch database (retrofit migrate --from root --to last, or from the latest Seal).
  • identify on a dump that matches no cut can take minutes, where a dump that matches answers in seconds. In one measured case, a history of about 120 operations, a dump matching no cut took 137 s against 0.57 s for one that matched.
  • drift over a database that skipped a migration the log records as a drop, as an out-of-order deploy or a migration-folder tool running an older pending migration does, re-adds the dropped column as an AddColumn, with no question and no note. Review a drift plan’s re-adds against the log before accepting them. A fix is planned after v0.10.0.
  • A released transition or cleanse data record is protected only by its position in the log: an edit to one behind a Seal passes validate, and migrate emits the edited bytes. Nor is a record the release shipped told apart from a branch’s backfill integrated behind it, which a database already past that point never runs.
  • Before 1.0 the oplog may gain operators and header keys. A build refuses a log with a newer retrofit-oplog-format or a key it does not know, so CI and every developer upgrade together. Planned after this release: op record; Archive, Unarchive and Purge; Annotate and Retract; erasing a released record; renaming a schema; possibly a marker for experimental operators.

Published .