Changelog
All notable changes to retrofit are recorded here, newest first, in the categories of Keep a Changelog. Before 1.0 retrofit promises no compatibility: a change is disclosed here, not prevented.
[Unreleased] records what the builds since the last release changed; a
release renames it to its version. Its fixed Log format heading records
every change to the oplog’s vocabulary, parameters, headers or seal bytes:
what changed and what a user does, usually regenerating the log. CI fails a
commit that changes internal/oplog/catalog/*.golden.json,
internal/oplog/serialize/ outside its tests, or
internal/seal/testdata/leafbytes_*.golden but adds no item under that
heading itself (make logformat-check); with make hooks, the
commit-msg hook refuses it at git commit. A commit whose serializer edit
changes none of the vocabulary, parameters, headers or seal bytes may
instead end its message with a Log-format: unchanged trailer; a reader
refusing a log it used to accept is a format change. A change to either set
of goldens always needs the item.
[Unreleased]
Log format
[0.13.0]
Added
op revertrecords a rollback in the log. Aftermigrate --downhas rolled a database back, it appends the reversal that down computes: the ghost’s ops after the log’s last op, the reference leg’s records, a declared reverse as one raw script, and the data-loss notes. A database migrated from the log then gets the same result, and a later down knows what was undone;migratestill never writes the log. Each appended op recordsreason="undoes <opid>", naming the window op it undoes, unlessreason=<text>is given. Recording twice appends nothing, and recording after more ops land appends only the new reversal. The fork guard and thevalidategate run before the write, and--dry-runpreviews what would be appended. A revert that is not at the log’s tail refuses asrevert-not-at-tail, and one whose leading leg is off a Seal asrevert-leading-leg-unsealed, each with a hint naming the way out.migrate --downreads whatop revertrecorded and advances its range past it, so a window is not undone twice.- A reversal that re-creates a dropped table, column or other element (the
down’s ghost and
op revert) keeps the element’s ElementID when the re-create passes the same-element predicate, so the element’s history continues where a fresh id orphaned it. The mint is markedsame_element=reversal(see Log format), and--dry-runprints the mark.validaterefuses an unmarked mint of an id the log already knows, andop reviserefuses to edit a marked mint.op squashnets a drop and the mints that return its element as one group, andop sealreports a data loss a squash of its range would avoid. - A down over a
DropTableorDropColumnsays that the element it re-creates comes back without its rows, as are-created-emptyitem ofreverse-irreversible, beside the forward’sdata-loss-on-applyline. A paired table’s stand-in names each column that comes back empty. A forward window that re-creates a dropped element says it comes back empty too. schema database-at <cut>pins the cut the local database is at, in the self-ignored.retrofit/database-at;--clearremoves the schema’s line, and bare it prints the pin in force and where it came from.op add,importanddriftalso take the pin as--database-atorRETROFIT_DATABASE_AT, and judge apply-risk from it (see Changed). A pin file that exists but cannot be read refuses asio-errorrather than being rewritten.op add RenameTable,op add RenameColumn,import,driftandop reverttake--allow-readers-moved <old-name>, repeatable. It attests that the readers of the table or column with that qualified old name are handled by the time the rename applies, and withholdsapply-risk-rename-in-usefor that rename; every other rename still warns, and the warning’s hint names the flag. A value that answers no rename, including one a refused run never reached, is reported as having no effect.op add CastColumnraisesapply-risk-table-rewritewhen the cast makes PostgreSQL rewrite the table, naming the dependents rebuilt with it, a partition’s own and a name-less index included.op addandop reviserefuse asbody-resolvea name in a function, trigger, view or raw script body that neither the log nor the project’s declarations hold where the body binds it: a top-level script statement, an always-reachedDOstatement, aBEGIN ATOMICbody, a%TYPEor%ROWTYPEdeclaration, or a string-literal SQL body with no polymorphic argument. Where PostgreSQL binds the name only when the body runs (a PL/pgSQL statement, a polymorphic SQL body, a conditional branch), they warnbody-name-absentinstead. A name an operator took off the schema (MoveColumn,ExtractTable, a partition operator,DropTableand the like) is named in the refusal with that operator, and a held body citing it raisesbody-name-removed.importjudges every body at the end of its source, andvalidate --historyholds a range to the same check: a certain site fails, a conditional one isadvisory. Objects the log cannot see, such as an extension’s functions, are declared in[[external]]entries inretrofit.toml.op addwarnsschema-partition-tree-indexedfor aCreateIndexon a partition root or middle, and for aPartitionTableof a table that already holds an index:identifyskips an indexed partition tree, so it cannot place a database over one. Where that tree is the only difference left, thefrom-dump-no-match,from-dump-behindand--expectoutcomes say so and thatdriftfinds nothing else differing, and outside--expectsay to name the cut withmigrate --from. Outcomes, codes and exit statuses are unchanged.
Changed
op add,importanddriftjudge apply-risk over the window from the database’s starting state, asvalidate --fail-on-apply-riskdoes, rather than one op in isolation. The starting state is theschema database-atpin, else the latest Seal at or before the op, else root. A table created inside that window holds no rows when the database applies the op, so the empty-table codes (apply-risk-add-unique,apply-risk-add-not-nulland the like) are withheld on it, andapply-risk-rename-in-useis withheld on an old name the starting state does not hold. A line saying a code was withheld on an assumed starting state prints once per run. A schema in a fork keeps the per-op verdict.- A repeated
--schemaon any verb refuses asflag-invalid-value, where the last value won:op seal --schema a --schema bsealed onlyb.--from,--toand--atstill take the last value. validateand every verb that folds the registered schemas (import,drift,snapshot,identify,element list,element history) refuse two[[schema]]entries rooted at one SQL schema asname-conflict, naming both. Identical imports had bound a cross-schema view on some runs and refused it on others.identifyrefuses a log that breaks a replay rule with the rule’s own code and hint, asvalidate,migrateandsnapshotdo: arule-no-live-referenceslog gets thereplace-or-drop-functionrecourse where it was refusedrender-blocked-on-partial-forestwith no hint. A replay-classed entry this build cannot check refuses asunverifiable.op add CreatePartition payment public p1readspublicas the schema, asCreateTabledoes, where it boundname=publicandpartition_by=p1. A partition named like the schema spellsname=.op reviseof an op an unsealed Seal once covered says the op was sealed under that Seal, since unsealed, so a database placed there is a working copy. It said the op was never sealed and a database at the Seal was unaffected.- A
RunRawScriptbody kept opaque by a cast to a managed type gets a hint for the cast’s shape: drop a'literal'::app.kindcast where the literal meets a column of that type, or write one statement per arm of aCASEof string literals. schema createof a name already taken refuses with a hint namingretrofit init --config <dir>/retrofit.tomlfor a second project, andinitbelow a project it leaves alone notes that project on stderr.validateno longer reportsschema-stray-oplogfor an oplog inside a nested project, a directory with its ownretrofit.toml.- Wording: a scalar param spelled as a flag (
--referenced_table t) refusesunknown-keywith a hint spelling itreferenced_table=<value>; themissing-requiredhint gives the param’s help; theSplitTableandMergeTableexamples spell--condition=; theRunRawScriptsummary says when a body is bound;data scaffold --purposesays both values are experimental; and the run-onceRunRawScriptnotes ofsnapshotandidentifyread in the singular for one op.
Fixed
op unsealno longer refuses a log for a verdict the log already held before the unseal, such as a sealed drop a newer replay rule refuses. It reopens the tail so the op can be repaired and the log re-sealed; the other verbs still refuse the log until then.op unseal,op removeandop reviseunder--dry-runno longer say the write stands when the post-write sweep refuses: the hint says nothing was written.- A function whose argument,
OUT, table or return type is a managed type or a relation’s row type, stored unqualified (f(mood)), is emitted qualified inCREATE,DROP,ALTER ... RENAME,COMMENT ONand the restored grants.snapshotandmigraterendered it bare, so theCREATEfailed or bound a same-named type on thesearch_path. Built-in and unmanaged types stay as written, andidentifyresolves a dump’s qualified spelling against such a function. driftno longer plans aDropFunctionand aCreateFunctionfor an unchanged function whose signature pg_dump prints differently from the log: a managed type schema-qualified (SETOF, arrays, row types andOUTcolumns included), a long-form type (timestamp without time zone,int), or a typmod PostgreSQL discards (varchar(10),numeric(10,2)). NewCreateFunctionrecords store the folded spelling; sealed records keep their bytes and still pair.op addrefuses asrule-unique-name-inaCreateFunctionPostgreSQL would read as an existing function: one that differs only by a discarded typmod (f(varchar(20))besidef(varchar(10))) or by qualifying its own schema’s managed type (f(mood)besidef(public.mood)). Both were stored, andmigratethen issued a secondCREATE FUNCTIONthat failed.driftandidentifyno longer plan aReplaceFunction, or answerfrom-dump-no-match, for aLANGUAGE sqlbody the log rendered whose string literal sits under aCASE, an array subscript or anarray_to_stringseparator. TheCASEand subscript fix reaches CHECK, index predicate, view and policy bodies too.identifyplaces a database whose skipped run holds a function body that reads nothing (BEGIN RETURN 2; END) ahead of a laterChangePrimaryKey,CastColumn,AddEnumValueorRenameTable, and a body whose only mention of a renamed table is anEXECUTEstring. It answeredfrom-dump-no-matchand routed todrift.identifyon a dump that matches no cut no longer runs the full placement search over a log holdingRenameIndex, where one such op took nearly a minute.identifyno longer refuses the project’s own snapshot when[[exclude]]names a schema an extension is placed in, and no longer scans a large snapshot in quadratic time per candidate cut.- A log whose first op names a
prev=the file does not hold no longer panicsmigrate,snapshotoridentify: they replay it alone and judge its replay rules.validateandop sealstill refuse it asstructural-prev-unknown. With several registered schemas,snapshot,element listandmigrateno longer refuse such a log asunverifiablewith a hint naming a build that does not exist. import --forcethat re-mints a schema’s ids warns for each other registered schema whose bodies still name the old ids, andvalidatethen failsrule-exists-and-alive.snapshotrefuses such a body asinvalid-cut, naming its holder, where it wrote an<elem:...>placeholder into the SQL.DropColumnof a columnExtractTablederived (a rejoin column, or a survivor’s foreign key column) refuses under its live dependents as on any table, where it was admitted and left a snapshot with aPRIMARY KEYon a missing column; drop the foreign key and primary key first.AttachPartitionover such keys no longer failssnapshotwithinvalid-cut.op reviserefuses a changedRunRawScriptbody thatop addwould refuse (a bare statement, two statements, an empty body), in every form. It wrote them at exit 0, formigrateto emit outside the one-DOcontract.schema removeremoves the schema’s block wherever it sits inretrofit.toml, in every spelling the loader accepts. It deleted the oplog at exit 0 but left the block when another[[schema]]followed. A removal that cannot take every registration refuses before the oplog is deleted.schema removeandschema movehonor an absoluteoplog_root:removedeleted a path that did not exist and left the real oplog, andmoverefused asmove-failed.schema moveandschema enginerewrite the config line the loader reads, where a quoted or repeated key reported success and kept the old value.- The
apply-risk-recorded-unsealednote advisesretrofit validate --fail-on-apply-risk --from <cut>; the command it printed passed--schema, whichvalidatedoes not take, and exited 2. op infono longer saysDropGeneratedExpressionends an element: the column and its values stay.
Log format
- The 15 operators that create an element (
CreateTable,AddColumn,AddPrimaryKey,AddForeignKey,AddUnique,AddExclude,AddCheck,CreateIndex,CreateView,CreateMaterializedView,CreatePolicy,CreateType,CreateFunction,CreateTrigger,CreateExtension) take one optional derived key,same_element=reversal, on a mint that returns an element a drop ended under its own ElementID. A line without the key is unchanged; no log needs regenerating. A build that predates the key refuses a line that has it, by name. - A value outside the declared set of a derived enum (
same_element, andMoveColumn’sarm) now reads as an unknown key, sovalidatereports the line unverifiable and the verbs that render or write refuse it by key, where it was akind-mismatchparse error. An authored enum still refuses.
Known issues
- Ending a column that a later
RunRawScriptbody names is refused only when the body is bound. For a body stored opaque,op removeorop reviseof the column, or a merge whose other side removed it, writes the log, andvalidateaccepts it.migratethen emits the script against a table without the column, so the migration fails whole at apply.validate --historyfails a pull request that adds such a body, unless the name sits only in a conditional branch or the body does not parse; it only advises on a body already on the target that the pull request leaves naming a removed column. Apply the migration to a scratch database in CI (retrofit migrate --from root --to last, or from the latest Seal). identifyon a dump that matches no cut can take minutes on a log holding aMergeTable,JoinTable,UnjoinTableor partition operator, where a dump that matches answers in seconds.driftasks thereversalquestion only when the database holds something logged after the drop. A database that has not yet run the log’s newest drop reads as intent, and the plan re-adds the item with no question. Review a drift plan’s re-adds against the log before accepting them.- In a project with several registered schemas,
driftplans aDropForeignKeyfor an unchanged cross-schema foreign key thatimportrecorded, skips the file’s key asadd-foreign-key-cross-schema-unsupported, and exits 0. Review eachDropForeignKeyin such a plan with--dry-runbefore recording it. op add,op sealandmigrateread only their own schema’s log, so a drop in one registered schema is accepted, sealed and emitted while another registered schema depends on its target.validatefails such a drop when the dependent is a view, policy, function or other body in another schema, but accepts it when the dependent is a foreign key or a column typed by the dropped type. Apply the migration to a scratch database holding every schema before it lands.migraterefuses asemit-render-faileda body that names an element of another registered schema, thoughsnapshotrenders it. The detail reads as if the element were dropped; it is live in the other schema’s log. No cut renders it. Usesnapshotfor a fresh database.- A trigger whose
EXECUTE FUNCTIONnames a function in another registered schema imports with afunction-synthesized-stubfinding and renders the call without its schema, which fails at apply or binds a same-named function on the search path. Treat each such finding as a broken trigger. - In a project with several registered schemas,
validate --fail-on-apply-risk --from <opid>resolves the opid in every schema, so the schemas that do not hold it refuse asopid-not-found. Name the cut with an anchor, such as a Seal, instead. driftrecords aCreateFunctionwhose signature names a type it set aside, so the function fails at apply under--allow-capability-gaps. Record the type before runningdrift.migrate --from rootof a log imported from apg_dump --schema-onlyfile can create a function ahead of the table its string-literalLANGUAGE sqlbody or%TYPEdeclaration reads, which PostgreSQL refuses under the defaultcheck_function_bodies.snapshotof the same log applies. Usesnapshotfor a fresh database.- A released
transitionorcleansedata record, in a project that opts in to them, is protected only by its position in the log: an edit to one behind a Seal passesvalidate, andmigrateemits the edited bytes. Nor is a record the release shipped told apart from a branch’s backfill integrated behind it, which a database already past that point never runs. - Before 1.0 the oplog may gain operators and header keys. A build refuses
a log with a newer
retrofit-oplog-formator a key it does not know, so CI and every developer upgrade together.
[0.12.0]
Added
driftover a database that skipped a logged drop, as an out-of-order deploy does, asks areversalquestion where it re-added the item with no question. The plan line names the op it reverses and that op’s Seal. The answers arekeep, which an unanswered question takes and which leaves the log as it is, andaccept, which records the restore. A table, a column, a key, a constraint, a trigger and a rename back all take the question.- A
RunRawScriptDObody that names only elements its binder can follow is now recorded bound to them. A later rename renders through the body inmigrate,op show, the down andreconcile’s diff, andop removeof an element the body names refuses asrule-exists-and-alive. A body the binder cannot read stays opaque. - The script
migrateemits reads the privileges held on a table, view or column before a re-create drops it and restores them on the replacement, so a grant made outside the log stays. Columns moved byMoveColumn,ExtractTable,JoinTableorInlineTableare named as not restored. driftandimportwithout--sealprint oneapply-risk-recorded-unsealednote per run, naming the tables, columns and renames they recorded: apply-risk reads what the unsealed tail creates as new and empty, and a table recorded from a populated database is neither.- The docs gain a refusal codes page, generated from the binary’s code registry, with each code’s kind, its exit statuses and the commands that can print it.
Changed
migraterenders aReplaceViewthe old way, oneCREATE OR REPLACE VIEW, only when the view’s column names at the op’s input cut are a name prefix of the new list and every prefix column’s type is proven unchanged. Any other definition is a rebuild: it drops the view’s tracked readers and the view, creates the view with the new definition, restores its grants, and re-creates its triggers and readers, with a-- WARNING:line naming each reader that may fail and a-- post-deploy:REFRESH for each materialized view re-created empty. The rebuild takes ACCESS EXCLUSIVE on the view and each reader, andop addraisesapply-risk-view-rebuildfor it. A view whose prefix column reads another view’s or a materialized view’s output column now rebuilds even for a tail-column add, since the type cannot be proven. In the lendlog fixture the two opsb56bc83a6cafand1ed10791196frender as rebuilds where they rendered oneCREATE OR REPLACE VIEWbefore;driftand the down emit oneReplaceView, no longer aDropViewand aCreateView, so the view keeps its ElementID.validate --historygains theopaque-bodycheck: it fails when the pull request adds a body retrofit stores opaque (aRunRawScriptwith nobound_params=, a string-literal or unresolvedLANGUAGE sqlfunction body, or a view, materialized view or expression slot whose text names no element), since a later rename or drop leaves what it names stale and only a database apply checks it. Each finding names the line, itsoperator,slotandcause, and the rewrite. A pipeline that keeps applying the range to a database before it lands adds--allow-database-apply: the check then lists its findings asadvisoryand does not fail.op add,importandop reviserefuse nothing new; the cause finding they print for such a body names the check and the consent, andimportnow prints it for a string-literal function and a view whose query falls back.reconcileno longer applies a forced order that the run did not ask for. It refuses asfork-forced-order, exit 1, naming the flag that asks for the order and what the order overrides, and the JSON error gainsforced_orders. A barereconcilereports each forced order.migrateover aRenameColumn,RenameTable,RenameVieworRenameMaterializedViewthat a plainLANGUAGE plpgsqlfunction reads follows theALTERwith aCREATE OR REPLACE FUNCTIONunder the new name, since PostgreSQL never rewrites such a body. Where the same migration also renames a type in the function’s signature, it refuses and names the drop-before, create-after sequence.driftno longer records a column whose type it set aside. A column typed by aCREATE TYPEthe log lacks is acolumn-type-unmintedskip, of class gap, in place of anAddColumnorCastColumnthat failed at apply.driftraises a merge question only between versioned or snapshot names, so a table prefix such aswp_no longer makes every table one family. It asks an index rename only between equal definitions, and pairs rename questions the same way on every import of one file.importanddata captureprintpreamble: N statementsbefore their summary or verdict, so the summary is the last line of stdout. The text, the JSON placement and the counts are unchanged.validate --fail-on-apply-riskandmigrate’s counsel count a table created in the window as populated once aMergeTable, aMoveColumndown, a splice-band data body or a staged load gives it rows. A range that passed may now exit non-zero.spec_refinop infoand theSPECline ofop add <Op> --helpprint the Operators page URL, not a repository path. Help text names docs pages by title and says operator, not SMO.
Fixed
- A rename now follows, and a drop now counts, the references in a function
body that earlier builds left stale or did not see: quoted, three-part and
qualifier names in
LANGUAGE plpgsql,%typedeclarations, reads through a CTE or an aliased subquery,THENresults,INlists,VALUESrows, row-SETsources and subscripts.RETURN QUERYandRETURN NEXTbodies and assignments to a parameter now bind. A read the binder cannot resolve makes the body opaque, where it had bound a shadowing variable or nothing. - A
BEGIN ATOMICbody’sINSERT,UPDATE,DELETEandMERGEstatements now bind, with eachMERGEclause scoped as PostgreSQL scopes it.driftno longer proposes aReplaceFunctionfor an unchanged one on every run. - A trigger’s
UPDATE OFlist and a bareWHENor parameterDEFAULTnow follow renames. TheUPDATE OFcolumns are quoted, anddriftorders a column drop after the triggers that list it. CreateViewandReplaceViewexpand a*inside a CTE, aFROMsubquery and every arm of a set operation, not only the top-level leftmost one. A rename that makes a derived relation’s read ambiguous now skips the view insnapshot, named, rather than writing one PostgreSQL rejects.migrate --downnow re-creates a droppedLANGUAGE plpgsqlfunction and a dropped trigger under the target’s names, orders a re-created function at its own position, drops a window-created reader ahead of the column it reads, and renders a bound script’s bare relations under the declared search path or refuses.- A reference floor crossing its declaration’s removal now applies as a
counted row delta when no key identifies it, writes a
body=nonefloor as its empty state’s delta, deletes a dropped NULL key NULL-safely, and folds a partition child’s columns by its root’s ids. A keyless floor had duplicated its rows at every apply, at exit 0. CopyTableandSplitTableclones of keys and indexes now render named. PostgreSQL had picked a numbered name when the derived one was held, and a laterDropIndexorRenameIndexthen hit the wrong object.importkeeps%TYPEon a function’s argument or return type and keeps a managed type named like a built-in’s internal name (public.bool). A function’sSETandRESETclauses fold as PostgreSQL’sproconfigdoes, and each element of itssearch_pathlist renders as its own literal.driftresolvesSET search_path FROM CURRENTasimportdoes.migrateparenthesizes a column default theDEFAULTslot cannot read bare, such asnow() AT TIME ZONE 'utc'.importmoves a drop past a later statement of the same file that clears theLANGUAGE plpgsqlreader blocking it, as PostgreSQL applies it.ops upgrade-sql-bodieskeeps a function’s volatility, parallel safety,SECURITY DEFINER,SET,COST,STRICT,ROWSandLEAKPROOF, which it had reset to their defaults while reporting success. A body given toop revisenow binds at the line’s cut, as one given toop adddoes.driftorders a merge or join question after a surviving foreign key’s drop, names the operator on a structural question’s text line where it printed<?>, and reports an earlier op that replay refuses under that op’s own code.validate --historyno longer failsfate-kepton a revise of a revision the base holds or on a removal whose replacer stays on the chain. Itscollapse-parentfinding names the revert and a recovery that completes after a hand edit rewires a successor.validatereports its other findings beside a seal digest mismatch.reconcilenames a remedy for a displaced identity generation, and describes a partition bound as the DDL spells it and an owned sequence by the name its column renders.identifyon a dump matching no cut no longer searches every placement on a log holding aCopyTable,SplitTableorExtractTable, or on a dump of an indexed partition tree. One such dump had run past 30 minutes.op shownotes the sealed ops its default scope hides, so a touching query over sealed history no longer reads as no history.data listmarks an orphaned record asdata checkrefuses it.- A refusal or hint no longer names a verb that refuses in turn, a flag the
user did not pass, or a generic cause.
op addandop reviselist the accepted keys for an unknownkey=value,importof onlyCREATE SCHEMAnamesschema create, a stream source prints as a bare-in recipes, a malformed foreign key target refuses under a registered code, and-vfolds identical repair sites into one line with a count. import --helpsays sources are never re-sorted and a glob expands lexically (V10beforeV2), with--from-filefor any other order.drifton a 4,651-op dry run takes 31 s, where it took 3,619 s.validateon a log whose tables churn columns grows about 8x at 4x the operations, where it grew 16.2x. A three-filereconcileover a 2,311-op log peaks at 262 MB, where it peaked at 1.82 GB.
Log format
ReplaceViewaccepts any new column list: replay no longer refuses a narrowing or a reorder, and theview-replace-supersetcode leaves the catalog, the registry and the rule census. A sealedReplaceViewwrites the same bytes and replays as before; the render now rebuilds the view whereCREATE OR REPLACE VIEWcannot take the change. An older build refuses such a line asview-replace-superset; upgrade it to read the log.- A
RunRawScriptwhose bound body names a quoted relation or column (app."Account","isActive") is now bound, where earlier builds kept it opaque. Its bound sentinel writes each quote of the name doubled, and the line gains a new producer-onlybound_quoted=[body]besidebound_params=, listing the body params that hold such a sentinel. An older build refuses the line by that unknown key, and itsvalidaterefuses it asbody-bound-edited; upgrade it to read the log. A rename of such a name to one that needs quoting now renders it quoted instead of refusingbody-render-unquotable. A line without a quoted name is unchanged, so a log needs no regeneration. - A bound body whose sentinel was hand-edited malformed now loads whether or
not
bound_quoted=lists it, and replay refuses it asbody-bound-edited; earlier in this release it refused at load asbody-bound-unmarked. - A plain
LANGUAGE plpgsqlfunction body now records the tables, views and columns it reads, as anansi-sqlbody does, sosnapshotwrites it with a renamed element’s new name. ADropTable,DropVieworDropColumnof an element such a body still reads refuses asrule-no-live-references, naming the function, atop add, where earlier releases accepted it. A log holding one is refused by that code wherever it is replayed to be checked or rendered (validate,snapshot,migrate,data check,data listanddata witnessamong them);op showandschema enginestill read it.data checkstill reads the sealed part of a log whose only such drop sits after its last Seal. The log’s bytes are unchanged. To read such a log, replace the function with a body that no longer reads the element (ReplaceFunction) or drop it (DropFunction) ahead of the drop, then regenerate the log or re-seal. - A trigger’s
WHENand a function parameter’sDEFAULTnow record the managed functions they call, as PostgreSQL does. ADropFunctionof a function either still calls refuses asrule-no-live-references, naming the trigger or the function, atop add, where earlier releases accepted it and the migration failed at apply. A log holding one is refused by that code wherever it is replayed to be checked or rendered. The log’s bytes are unchanged. To read such a log, drop the trigger (DropTrigger), or replace the calling function without the call (ReplaceFunction) or drop it (DropFunction), ahead of the drop, then regenerate the log or re-seal. A call to one of two overloads of a name, or an unqualified call whose name a built-in shares, still records nothing. RunRawScriptgains the optional, producer-writtenbound_params=[body, reverse_body]param, written after the reversal params and beforereason. A build without it refuses a line that holds it as an unknown key; no log written today holds it, so nothing needs regenerating.- A body param whose text holds the bound-script form
"<bound:…>"but thatbound_params=does not list refuses at load asbody-bound-unmarked, whatever the operator. A log written by retrofit never holds one. Restore the marker, or replace each"<bound:…>"with the name it stands for. - Replay refuses more drops PostgreSQL refuses, as
rule-no-live-references, atop addand wherever a log holding one is replayed: aDropFunctionwhile a partition key calls the function; aDropColumnorMoveColumnof a column a trigger’sUPDATE OFlist names; aDropTablewhile a column is typed by its row type or a body names that type; and a drop of what aBEGIN ATOMICbody’s DML statement, a subscript or a%typedeclaration reads. The log’s bytes are unchanged. To read such a log, clear the reference ahead of the drop, then regenerate the log or re-seal. - A body that names a dropped element refuses as
rule-exists-and-alive, for a view, materialized view, function, policy, triggerWHEN, column default, generation expression or partition key expression. Only a merge that orders the drop first reaches this, andreconcile --cleannow refuses that order asfork-no-clean. - An index, or a
PRIMARY KEY,UNIQUEorEXCLUDEconstraint, named as another relation or index of its schema refuses asrule-unique-name-in. It failed at apply withrelation ... already exists. validateandop sealrefuse aprev=cycle asstructural-prev-cycleand a danglingprev=asstructural-prev-unknown, exit 1, wherever in the file they stand. A merge resolution or a rebase can leave either.- A
CopyTableorSplitTablethat clones a key or index whose name cannot be derived refuses ascopy-clone-name-collision, naming the workaround of dropping it first and re-adding it after. TheEXCLUDEcase had refused asrule-exists-and-alive.
Known issues
- Ending a column that a later
RunRawScriptbody names is refused only when the body is bound. For a body stored opaque,op removeorop reviseof the column, or a merge whose other side removed it, writes the log, andvalidateaccepts it.migratethen emits the script against a table without the column, so the migration fails whole at apply.validate --historyfails a pull request that adds such a body. A pipeline that passes--allow-database-applyshould apply the migration to a scratch database (retrofit migrate --from root --to last, or from the latest Seal). identifyon a dump that matches no cut can take minutes on a log holding aMergeTable,JoinTable,UnjoinTableor partition operator, where a dump that matches answers in seconds.driftasks thereversalquestion only when the database holds something logged after the drop. A database that has not yet run the log’s newest drop reads as intent, and the plan re-adds the item with no question. Review a drift plan’s re-adds against the log before accepting them.- In a project with several registered schemas,
driftplans aDropForeignKeyfor an unchanged cross-schema foreign key thatimportrecorded, skips the file’s key asadd-foreign-key-cross-schema-unsupported, and exits 0. Review eachDropForeignKeyin such a plan with--dry-runbefore recording it. op add,validate,op sealandmigrateread only their own schema’s log, so a drop in one registered schema is accepted, sealed and emitted while another registered schema depends on its target. Apply the migration to a scratch database holding every schema before it lands.migraterefuses asemit-render-faileda body that names an element of another registered schema, thoughsnapshotrenders it. The detail reads as if the element were dropped; it is live in the other schema’s log. No cut renders it. Usesnapshotfor a fresh database.- A trigger whose
EXECUTE FUNCTIONnames a function in another registered schema imports with afunction-synthesized-stubfinding and renders the call without its schema, which fails at apply or binds a same-named function on the search path. Treat each such finding as a broken trigger. driftrecords aCreateFunctionwhose signature names a type it set aside, so the function fails at apply under--allow-capability-gaps. Record the type before runningdrift.- A released
transitionorcleansedata record, in a project that opts in to them, is protected only by its position in the log: an edit to one behind a Seal passesvalidate, andmigrateemits the edited bytes. Nor is a record the release shipped told apart from a branch’s backfill integrated behind it, which a database already past that point never runs. - Before 1.0 the oplog may gain operators and header keys. A build refuses
a log with a newer
retrofit-oplog-formator a key it does not know, so CI and every developer upgrade together.
[0.11.0]
Added
op info --as json, for one operator or the--dump, gives each operator averbsobject: forop_add,import,drift,snapshotandmigrate, whether that verb ships the operator. Each value isavailable,unavailable, orsyntax-unavailablefor an operator this build cannot spell.
Changed
- A fresh text-mode
snapshotloads only the rows a record states as its table’s whole content (areplacesection, or a referencerestate) between the tables and the constraints, as before, and fires no trigger on them. Every other record (anappendordeltasection, a seedrestate) now runs in a second-- retrofit:snapshot-data v1band after the whole schema, beforeCOMMIT, with constraints, indexes and triggers live, asmigrate --from rootruns it, so its rows fire the enabled triggers and meet the foreign keys in authored order. Each band re-seats its own tables’ sequences. A note,snapshot-state-under-trigger, names each state table and triggermigratewould fire; a warning,data-statement-order-fk, names a foreign key the order meets (a state ahead of its parent’s statements, or a child’s statements ahead of its parent’s) with the recourse.--as=json,--data-onlyand the banner opid are unchanged. migratewarnsdata-statement-order-fkfor the same two orders in its trailing seed band, where every foreign key is live: a stagedreplacetable whose key reaches a parent only the band’s later records supply, and a child’s records ahead of its parent’s. The warning names the table, its parent, the key and the recourse; a key declaredINITIALLY DEFERREDraises neither, except under--envelope=independent. The script is unchanged.- The
drift plan:header counts changes, between removals and questions:drift plan: N additions, M removals, C changes, Q questions. A change is any planned op the header does not count as an addition or a removal, such as a rename, a cast or a NOT NULL flip, so a plan made only of those no longer reads0 additions, 0 removals. The JSONplan-headerrecord gains achangesfield. transitionandcleansedata records are experimental. Every verb refuses them unlessretrofit.tomlopts in ([experimental] purposes = ["transition", "cleanse"]), and warns on every use once it does: a released one is protected only by its position in the log. Write a backfill as aDOblock to keep it under a Seal’s digest. Without the key,import --purposeanddata scaffold --purposerefuse to mint either, andvalidate,data check,data list,data witness,data repin,migrateandsnapshotrefuse a corpus holding one, asexperimental-not-enabled, naming each file and its purpose. With it, each verb reads and mints as before and warnsexperimental-purposeonce per such file. The table is project-wide; an entry naming anything else refuses asconfig-invalid. A retrofit that predates the table refuses the whole file (unsupported table [experimental]).migrate --downreports a row loss its render chose (data-loss-on-apply-render-chosen) as a warning, asdata-loss-on-applyis, not as a note after the other warnings.migrate --allow-unreached-dml, removed in 0.10.0, refusesflag-retiredand says to drop the flag, instead of failing as an unknown flag.op show --touchingresolves a function, trigger, policy, extension or materialized view, by name, signature or element id, aselement historydoes.- Refusals that pointed at generic advice now name their own recovery:
a bare token in a payload record names the record’s fields (and
ExtractTable’s
--into); a body calling an overloaded function says no spelling binds it; a data file refused only for its record marker line leads with the marker grammar;reconcile --cleanover a demotion whose other order replays clean names that order’s flags before--accept-dirty;schema engine <name> <version>names--schema. - A skipped sequence names itself on drift’s skip line and
skippedevent, and identify’s residual skip lines no longer end in a space. after-traffic-outcomeon aRunRawScript, whose declared reverse is opaque, says the down may refuse or may drop the rows written since, rather than that it does.snapshot --help-alland docs/snapshot.md say that triggers and policies follow the data band, so a snapshot build runs no trigger on its seed rows whilemigrate --from root --labeldoes.
Fixed
- The
drift plan:header counts the creation of a policy, function, trigger or extension as an addition and its drop as a removal, as it does for a table or a view; it had counted them as changes.--show=removalsnow hides those creations and--show=additionsthose drops. migrateno longer emits a script PostgreSQL refuses with55006(pending trigger events) when an op writes a table holding aDEFERRABLE INITIALLY DEFERREDconstraint trigger and the same transaction then alters that table, asAddNotNull … fill=does. Under--envelope ownit refuses asmigrate-deferred-trigger-pending, naming the trigger, the table and the ops; undercallerthe script and adeferred-trigger-owedwarning name theSET CONSTRAINTS <schema>.<trigger> IMMEDIATE;the enclosing transaction can run first. A--downis treated the same way.migratenow warns, asdeferred-trigger-pending, where a data record writes a table holding aDEFERRABLE INITIALLY DEFERREDconstraint trigger and a later op of the same transaction alters that table, which PostgreSQL refuses with55006. The script has a-- retrofit:line above that op, and the warning names theSET CONSTRAINTS <schema>.<trigger> IMMEDIATE;the record can end with.op seal,import --sealanddrift --sealnow refuse, asdata-file-anchor-unresolved(exit 2), while the schema’s data corpus holds a record whoseanchor-op=no longer names an op, asdata checkdoes, with the same detail anddata repin’s recourse. Before, the seal was written,migraterefused the release it cut, and repairing the record neededop unsealfirst, sincedata repinrefuses a sealed target. Regenerating a log under a committed data file, orop reviseorop removeof the op a record anchors on, leaves such a record.data checknow refuses an oplog whoseretrofit-oplog-formatline is older or newer than this build reads, asoplog-format-too-oldoroplog-format-too-new, exit 1. Before, it reported no findings and exited 0 on a log every other verb refuses, so a check run alone passed it. On a sealed log,validatenow refuses the same line by the same code and the parser’s own hint, notsealed-oplog-unparsablewith advice to repair the line.op add PartitionTable,CreatePartitionandAttachPartitionnow warnschema-partition-bound-timestamptz-zonelesswhen a range or list bound on atimestamptzkey has no UTC offset (from=2024-01-01). PostgreSQL reads such a bound in the applying session’s time zone, soidentifyanddriftcannot compare it with pg_dump’s output. The op still appends; the warning names the zoned spelling (2024-01-01 00:00:00+00).identify’s refusal detail now gives each skip’s hint under it, so a zoneless bound reads as needing an offset rather than as unsupported partitioning.import --dry-runnaming a schema that is not registered no longer creates it. The run still previews the import, then leaves the project as it found it: no<schema>.oplog, no[[schema]]row, nodefault_schemapin, and no newoplog_rootdirectory. Previewing such an import no longer needs aretrofit schema createfirst.snapshot --include-drop=scopednow drops the views and materialized views it writes, before the tables, so it re-applies to a database that already holds a schema with a view over one of its tables. Before, the script stopped atDROP TABLEbecause the view depended on the table, and rolled back. An object the prologue does not drop, such as a function or a view outside the snapshot, still fails the apply when it depends on a table or view the prologue drops.importofDROP MATERIALIZED VIEW IF EXISTSnaming a materialized view the import does not hold is now a counted no-op that exits 0, asDROP VIEW IF EXISTSis. Before, it was a capability gap and exited 2.IF EXISTSnaming a table, view or index still exits 2, since PostgreSQL refuses it as not a materialized view.snapshot --include-drop=scopedof a schema with a foreign key now applies to an empty database. The prologue’s foreign key drop readsALTER TABLE IF EXISTS … DROP CONSTRAINT IF EXISTS …, so it does nothing when the table or its schema is absent. Before, it failed withschema "<name>" does not existorrelation "<name>" does not existand the script rolled back.
Log format
- A data file whose header says
purpose=transitionorpurpose=cleanserefuses asexperimental-not-enabledon every verb that reads it, unlessretrofit.tomlopts in with[experimental] purposes = ["transition", "cleanse"]. The file’s bytes and the oplog are unchanged; add the key to read such a log as 0.10.0 did, with a warning per file.
Known issues
- Ending a column that a later
RunRawScriptbody names is not flagged:op removeorop reviseof the column, or a merge whose other side removed it, writes the log, andvalidateaccepts it.migratethen emits the script against a table without the column, so the migration fails whole at apply. To catch it before landing, have CI apply the migration to a scratch database (retrofit migrate --from root --to last, or from the latest Seal). identifyon a dump that matches no cut can take minutes, where a dump that matches answers in seconds. In one measured case, a history of about 120 operations, a dump matching no cut took 137 s against 0.57 s for one that matched.driftover a database that skipped a migration the log records as a drop, as an out-of-order deploy or a migration-folder tool running an older pending migration does, re-adds the dropped column as anAddColumn, with no question and no note. Review a drift plan’s re-adds against the log before accepting them.- A released
transitionorcleansedata record, in a project that opts in to them, is protected only by its position in the log: an edit to one behind a Seal passesvalidate, andmigrateemits the edited bytes. Nor is a record the release shipped told apart from a branch’s backfill integrated behind it, which a database already past that point never runs. - Before 1.0 the oplog may gain operators and header keys. A build refuses
a log with a newer
retrofit-oplog-formator a key it does not know, so CI and every developer upgrade together.
[0.10.0]
Added
- This change log. CI requires a “Log format” entry for any change to the
oplog catalog golden, the serializer or the seal’s leaf-byte goldens. A
serializer edit that leaves the format unchanged may say so with a
Log-format: unchangedcommit trailer instead. - Each release archive carries a man page per command and the bash, zsh
and fish completion scripts, and the Homebrew cask installs them, so
man retrofit-migrateworks afterbrew install. migrateandsnapshottake--allow-unnamed-label <label>(repeatable). It attests that a labelled dataset was left unnamed on purpose, such as a test-onlyfixtureslabel on a long-lived target. Theapply-risk-data-exclusionswarning no longer counts that dataset, and its hint names this flag beside--label. Each value names one exact label, so a dataset added later still warns. An attestation that answers nothing prints a note that it had no effect.CopyTableandSplitTablecarry eachEXCLUDEconstraint of their source onto every table they create, under the name PostgreSQL gives it there (<table>_<keys>_excl);migrateadds it after the table’sLIKE.ExtractTable,JoinTableandUnjoinTablemove anEXCLUDEunder its own name onto the table that receives its key columns, and refuse one whose keys would land on two tables, naming it.MergeTabledrops each source’sEXCLUDEwith the source, and a merged row that breaks one onintofails the apply. A copy refuses anEXCLUDEwhose name it cannot derive, or two that would take one name.InlineTablecarries anEXCLUDEconstraint where it refused a table holding one. The survivor keeps its own, and a source’s moves onto the survivor under its own name, re-keyed onto the survivor’s key columns, as aUNIQUEdoes.migratechecks the rows the re-added constraint would hold by its own operators and refuses, naming it, when a padded row, or on the lookup arm two survivor rows sharing a folded row, would conflict, before anything is dropped.migrate --downof anExtractTableover a table holding anEXCLUDEnow inverts, re-adding it on the table it came from.
Log format
AddUniquetakes an optionalnulls_not_distinct=truefor aUNIQUE NULLS NOT DISTINCT, whichimportnow carries in every spelling where it skipped the key asunique-constraint, anddriftwhere it skipped it asadd-unique-unmodeled-facet.snapshotandmigratewriteUNIQUE NULLS NOT DISTINCT (…).driftrecreates a key that differs from the target only by it (DropUnique, thenAddUnique), andidentifytells the two apart. Replay refuses the key besidebacking_index, sinceUSING INDEXtakes the index’s NULLS treatment, and akeepover such a key refuses withreverse-policy-not-null: the NULLs it writes still collide. AUNIQUEthat also carriesINCLUDE,WITH (…), a tablespace or an access method still skips. A build without the key refuses a log that holds it, naming the key. A log that skipped such a key keeps the skip; re-import the statement to carry it.- The
columnslist ofCreateView,ReplaceView,CreateMaterializedViewandReplaceMaterializedViewpercent-escapes each name’s%,,and;(%25,%2C,%3B), so a quoted column name holding a comma, such as"revenue, usd", stays one column where it split into two.import,driftandmigratewrite the escaped form;op add’scolumns=takes it too. A list whose names hold none of the three is unchanged, as is its content hash. A log that holds a literal%25,%2Cor%3Bin a view column name now reads it as%,,or;; re-import that view. No other log needs regenerating. CreateTriggertakes optionalconstraint=true,deferrable=trueandinitially_deferred=truefor aCREATE CONSTRAINT TRIGGER, whichimportanddriftnow carry where they skipped it astrigger-unsupported. ItsFROMform still skips astrigger-unsupported. Replay refuses a constraint trigger that is notAFTER,FOR EACH ROW, on a table and without transition tables,initially_deferredwithoutdeferrable, either one on a plain trigger, andRenameTriggerof a constraint trigger. Its name is also one of its table’s constraint names: aCHECKor key of that name refuses withrule-unique-name-in, and an unnamed one takes the next free derived name.importrecordsALTER TRIGGER … RENAMEof one as the gapconstraint-trigger-rename-unsupported, and refuses a trigger created over one’s name, or one created over a plain trigger’s (constraint-trigger-replace-invalid).snapshotandmigratewriteCREATE CONSTRAINT TRIGGERwith itsDEFERRABLEclause, andmigratecounts it among the constraints a deferred key’s name must not share. A build without the keys refuses a log that holds them, naming the key. No log needs regenerating.CreateFunctiontakes an optionalprocedure=truefor aCREATE PROCEDURE, whichimportanddriftnow carry where they skipped it ascreate-procedure-unsupported. A procedure shares the functions’ names and signatures; replay refusesreturns,volatility,parallel_safety,strict,cost,rowsorleakproofon one, and a trigger that executes one.snapshotandmigratewriteCREATE OR REPLACE PROCEDURE, and name it as aPROCEDUREinDROP,ALTER … RENAMEandCOMMENT ON, each argument type markedIN.importreads thePROCEDUREandROUTINEforms ofDROP,ALTERandCOMMENT ON; with theFUNCTIONform, a name written without an argument list names the one routine of that name and kind, of any arity. It refuses what PostgreSQL refuses: an unmarked argument list two procedures answer to, or a name without a list that more than one routine answers to (routine-name-not-unique), and a statement naming a routine of the other kind, aCREATE OR REPLACEacross kinds or a trigger on a procedure (routine-kind-mismatch). A build without the key refuses a log that holds it, naming the key. No log needs regenerating.CreateFunctionandReplaceFunctiontake optionalrows=(a function’sROWS n, kept as written, likecost=) andleakproof=.importanddriftcarry both fromCREATE FUNCTION, where they skipped the function asfunction-rows-unsupportedorfunction-leakproof-unsupported, andimportfolds anALTER FUNCTION … ROWSor… LEAKPROOFinto aReplaceFunction;driftstill skips a targetALTER FUNCTION, as it does… COST. A build without them refuses a log that holds them, naming the key. No log needs regenerating.- A log holding one of the PostgreSQL 18 surfaces (
AddColumn generated_kind=virtual,enforced=false,period=true,without_overlaps=true,SetConstraintEnforced,SetConstraintInherit) now failsvalidatewithsyntax-not-available, and no verb renders it. Regenerate the log without them, or revise each op to its PostgreSQL 17 form (generated_kind=stored,enforced=true,period=false,without_overlaps=false). RunRawScripttakesreverse_noop=<reason>andreverse_blocked=<reason>besidereverse_body, at most one per line, and a newDeclareReversalline (target=,supersedes=, one of those arms) declares a sealed script’s reversal without changing a sealed byte. A build without them refuses a log that holds them, naming the key or operator. No log needs regenerating.reconcile --base --ours --theirsreads an input that has noengine-versionheader, and merges the key like any other: an absent key is a value. Every other command still refuses such a log, and no log written before needs regenerating.reconcile --base --ours --theirskeeps a side’s own followers’prev=bytes when the other side removes the line they follow: the removed line stays in that side’s place as its tombstone. No log needs regenerating.- The three-file mode’s refusal hints name the inputs by role (ours, theirs) and its own re-run, never a flag it refuses.
reconcile --base --ours --theirsrefusesfork-write-writefor a value both sides change that no block order decides (one side revising a base op in place, the other setting the same property);--keepor--swapwrites the stated outcome. No log needs regenerating.- The conflict-marker and duplicate-opid refusals name the three-file mode
(
reconcile --base --ours --theirs --result) with its inputs by role, and a partition-scoped column facet set against its parent’s recursing op refusesfork-write-writenaming<partition>.<column>. No log needs regenerating. reconcile --base --ours --theirswrites--swap’s order with a block’s tombstones beside the block they follow, so--swapover (base, ours, theirs) writes what--keepover (base, theirs, ours) writes, and reads a line a side holds as the base does through the other side’s revision of its predecessor as unchanged. No log needs regenerating.AddIdentity,DropIdentityandSetIdentityGenerationtake an optionalpartition=, a partition beneath the column’s owner whose own identity the op adds, drops or changes in place on a PostgreSQL 15 or 16 pin.op addwrites it for a<partition>.<column>token there; from 17 on the op refuses on a partition’s column. No log needs regenerating.AddColumnandAddIdentitytake optionalidentity_start,identity_increment,identity_minvalue,identity_maxvalue,identity_cacheandidentity_cycle=truefor an identity’s sequence options, whichimportnow carries on an inline identity column,ADD COLUMNandADD GENERATEDwhere it skipped them asidentity-options-dropped. An option that restates PostgreSQL’s default for the column’s type and direction is dropped;AS,OWNED BY,RESTARTand a partition’s own identity still skip. Replay refuses an option without an identity (identity-option-without-identity), one that is not an integer (identity-option-not-integer), and a bound outside the column’s type,MINVALUEnot belowMAXVALUE, aSTARToutside them,INCREMENT0 orCACHEbelow 1 (identity-option-out-of-range).snapshotandmigratewrite the options in the identity’s sequence list;driftcarries the target’s options onto an identity it adopts and reportsidentity-options-differfor a paired one whose options differ. A build without the keys refuses a log that holds them, naming the key. A log that skipped such options keeps the skip; re-import the statement to carry them.SetIdentityOptionschanges an identity column’s sequence options (ALTER COLUMN … SET START WITH | INCREMENT BY | [NO] MINVALUE | [NO] MAXVALUE | CACHE | [NO] CYCLE). It takesidentity_start,identity_increment,identity_minvalue,identity_maxvalue,identity_cache,identity_no_minvalue,identity_no_maxvalueandidentity_cycle(falseisNO CYCLE); an option it does not name keeps its value.importwrites it for those forms, which it skipped asAlterTableSetIdentity(RESTARTstill skips), anddriftfor an identity whose options differ from the target’s, which it skipped asidentity-options-differ. A build without the operator refuses a log that holds it. No log needs regenerating.SetIdentityOptionstakes an optionalpartition=, asSetIdentityGenerationdoes, and a partition-scopedAddIdentitytakes theidentity_*options it refused: on a PostgreSQL 15 or 16 pin a partition’s own identity keeps its sequence options throughATTACHandDETACH, which dropped them.importwrites them for anADD GENERATEDorALTER COLUMN … SETon an attached partition, which it skipped asidentity-options-droppedorAlterTableSetIdentity, anddriftplans the scopedSetIdentityOptionsfor a partition whose own identity’s options differ from the target’s. A log that skipped such a statement keeps the skip; re-import it to carry the options.AddExcludeandDropExcludeadd and drop anEXCLUDEconstraint.AddExcludetakesmethod(btree,gist,spgistorhash), the key list asCreateIndexspells it (columns,key_expressions,key_opclasses,column_directions,nulls_ordering),key_operators(each key’sWITHoperator,&&orOPERATOR(schema.op)),name,deferrableandinitially.importwrites them forALTER TABLE … ADD CONSTRAINT … EXCLUDE, the table-levelEXCLUDEinsideCREATE TABLEandDROP CONSTRAINTof one, where it skipped the first asalter-table-otherand the second asinline-exclude-constraint. AnEXCLUDEwith aWHERE,INCLUDE,WITH (…),USING INDEX TABLESPACEor a keyCOLLATE, or on a partitioned table or a partition, still skips, now asexclude-constraintin both spellings.snapshotandmigratewrite it, andAddExcluderaisesapply-risk-add-exclude.InlineTable,PartitionTableandDepartitionTablerefuse a table that holds one, andAddExcluderefuses a partitioned table. A build without the operators refuses a log that holds them. A log that skipped anEXCLUDEkeeps the skip; re-import the statement to carry it.driftreads anEXCLUDEin both spellings and pairs it with the log’s by its keys, method, operators, operator classes, sort order and deferral, not by name. A difference in any of them plansDropExcludethenAddExclude, and a name-only difference plansRenameConstraint. An operator pg_dump qualifies by the search path’s first schema (OPERATOR(public.===)) compares equal to the bare===. A targetEXCLUDEwith a facetAddExcludecannot carry is anexclude-constraintskip and drops nothing.identifykeys the same facets, so two cuts differing only by anEXCLUDE’s operator are not one match. Before, drift reported everyEXCLUDEthe log held as anexclude-constraintskip and ignored the target’s.AddExcludetakes optionalinclude_columns,storage_paramsandpredicatefor anEXCLUDE’sINCLUDE (…),WITH (…)andWHERE (…), each spelled as onCreateIndex.importanddriftnow carry the three where they skipped the constraint asexclude-constraint, andsnapshotandmigratewrite them. An unnamedEXCLUDEwithINCLUDEtakes PostgreSQL’s name over its keys and itsINCLUDEcolumns (t_x_n_excl), whereimportheld its name over the keys alone. AWHEREthat does not resolve skips the whole constraint, since without it the constraint compares every row; aWITH (…)value the log cannot spell,USING INDEX TABLESPACEand a keyCOLLATEstill skip.driftplans a change to any of the three asDropExcludethenAddExclude, andDROP COLUMNof a column only theINCLUDEorWHEREnames drops the constraint with it, as PostgreSQL does. A build without the parameters refuses a log that uses them. A log that skipped such anEXCLUDEkeeps the skip; re-import the statement to carry it.PartitionTabletakes two optional lists,partition_parentsandpartition_keys, one entry per partition, to declare a tree of several levels; a single-level record is unchanged and carries neither. Nothing to regenerate: an existing log replays as it did.op show --framing=addwrites a partition bound or key as one JSON object, whichop addtakes back.- Every oplog names its form on a fourth header line,
# retrofit-oplog-form: mergeor# retrofit-oplog-form: compact. A log without the line is refused (oplog-form-header-missing), as is any other value (unknown-oplog-form). In the merge form, whichschema createwrites unless--form compactis given, a removed or revised operation stays in the file as a~tombstone or behind areplaces=list;op compactdrops those traces. Every read checks each live Seal over its chain, so a Seal whose digest no longer matches is refused. For an existing log, add# retrofit-oplog-form: compactas line 4, after# engine-version:, then runretrofit schema form mergeto keep the history from then on. op show’s runnable framings printCreateIndex’skey_opclassesas--key_opclass, not--key_opclasse. The log’s bytes are unchanged; re-runop showfor any copied--key_opclasseline.- A
# retrofit-*header key this build does not know is refused on every verb that loads the oplog (unknown-key,writes-blocked-on-unknownsorrender-blocked-on-unknownsby the verb’s role), naming the key and its line, instead of being carried through opaquely. Run a build that knows the header; do not delete it. ExtractTabledeclares the ruleextract-partitioned-source: an extract that moves every live column refuses a partitioned parent or a partition as its source. A log holding such an extract no longer replays. Regenerate the log, runningDepartitionTableorDetachPartitionbefore the extract, or leave a parent’s partition key on it so the source survives.AttachPartitionends the partition’s own columns into its parent’s, andDetachPartitionmints the detached table’s columns and CHECKs afresh, recording their IDs in its new optionalcolumn-idsandadjacent-ids.SetDefault,DropDefault,AddNotNull,DropNotNullandSetColumnCommenttake an optionalpartition(partition=<table>, or the on-column spelled<partition>.<column>) that scopes the op to one partition of the column’s family. New refusals:partition-not-beneath-owner,partition-not-null-inherited,partition-identity-absent,detach-id-list-arityandattach-partition-check-mismatch. A log whose ops name an attached partition’s own columns, or a detached table’s columns by their earlier IDs, no longer replays: regenerate any log that holds anAttachPartition.
Fixed
migrate --downover a window that drops a table holding anEXCLUDEconstraint, or merges such a table into another, re-creates the table with eachEXCLUDEunder its name. It refused the windowmigrate-non-reversiblewhere the same window over aUNIQUEinverted. AnEXCLUDEaCopyTableorSplitTablecloned still refuses. The down of anExtractTablethat inverts as anInlineTablestill refuses a table that holds anEXCLUDE, and now names the constraint.reconcile <fork-selector> --cleanon a fork nested in another fork’s block closes that fork alone. It also hung the enclosing fork’s other block onto the result, closing the enclosing fork in an order nothing checked, and when that block’s lines came first it moved the nested fork’s ops under it. The enclosing fork now stays a fork, which the nextreconcile --cleanwrites or refusesfork-no-clean. Thefork-nestedrefusal and the bare report’s next step name the nested fork’s selector to close first, where they said no reconcile invocation could resolve it.op add ChangePrimaryKeyrequiresname=for the new key, asAddPrimaryKeydoes, unless it setsbacking_index, anddriftandimportgive a rotated-in key PostgreSQL’s<table>_pkey. An unnamed key was added bare, so the name PostgreSQL gave it depended on where an earliermigratewindow ended, and a later window could drop or rename a name the database did not hold. A log that already holds an unnamedChangePrimaryKeystill reads.migratereports a seed record no--labelnamed asnot-selectedeven when it is anchored behind--from. It used to print adml-behind-windowwarning for it, and--allow-unnamed-labelnoted that it had no effect, so afixturesdataset kept warning on every later window. A selected record behind the window still warnsdml-behind-window.- The
dml-chain-near-misswarning a data mint raises compares a new chain only with existing chains of its own band (reference, seed or history).data capture --label fixtureson a table that also holds acleansechain no longer warns that a label drift forks the chain, or suggests re-running with the other chain’s label-set: a chain of another band is never the one a capture supersedes. data capture’s verdict line printsnot comparable:once before the reason instead of twice, anddata capture --dry-runrehearses a table it would capture aswould capture:, notwould captured:.migrate’sdata-replace-prologueanddata-replace-cascadewarnings say a replaced table “is replaced by its full observed state” andreplacing <table>, not “cleared” and “clearing”: a keyed replace upserts the observed rows and deletes only the keys the state lacks, and never empties the table.op add CreateIndex --helpspells the per-key operator-class flag--key_opclass, whichop addandop reviseaccept beside--key_opclasses. The misspelling--key_opclasseis refused.initagainst an existing.gitattributesprints a note naming each line it appends, instead of changing the file silently.- A data file’s record marker line that leaves out a required token
(
kind=,seq=or thesrc=provenance) is refused naming the missing token, not as carrying an “unrecognized or mis-ordered” one. Every marker-line refusal (G-e) now spells the marker grammar as its remedy, and the shared repair hint names the data file;data checkand the other corpus readers no longer append a line about a “sidecar”. identifyandmigrate --from-dumprefusefrom-dump-schema-absentfor a dump of a database that holds neither the (non-public) schema nor any of its objects, and point atmigrate --from root. They matched such a dump to theCreateSchemacut, somigrate --from-dumpemitted a script withoutCREATE SCHEMAthat failed when applied.identifyandmigrate --from-dumplist the schema facets that did not pair (cut only:anddump only:lines in the detail) when the dump matches no cut butdriftwould plan nothing against the nearest one. Thefrom-dump-no-matchhint no longer sends the user to adrift --dry-runthat reports 0 ops, and thefrom-dump-behindhint no longer says such a divergence was made outside the log; both offer the nearest cut by hand withmigrate --from.retrofit completion <shell>prints the shell’s completion script. The root’s help listed it, but running it was refused as an unknown command, and the scripts’ completion requests were refused the same way.- Every verb that loads the oplog refuses an operator this build does not
know, a
~tombstone included, naming the operator and the line, with the hint “upgrade the CLI to one that knows the operator; do not delete or edit the line”. Theopverbs andvalidaterefuse it asunknown-operator, except thatvalidatekeepssealed-oplog-unparsablefor one inside a sealed range. A renamed or split operator keeps its re-import or rename remedy. validatereports an unknown key once, naming its line, the key and the rules it leaves unverifiable, instead of oneunverifiablerow per rule for every op of the operator. No hint says to hand-fix the op. It also reports a header key this build does not know, and aretrofit-model-versionnewer than the build, as unverifiable (exit 2); an unsealed log with a newer model-version no longer passes.reconcileexits 2, not 1, for a file only a newer build can read: an unknown operator, a too-newretrofit-oplog-formatorretrofit-model-version, an unknown header, and (as before) an unknown key. The JSONcodetells them apart; a conflict marker, a repeated header or a refused fork stays exit 1.reconcile, theopverbs andop revise -irefuse an oplog that exists but cannot be read (for example mode 000) withoplog-unreadable(exit 2), printingcannot read oplogand the read error, instead ofoplog-not-found(“no oplog”) or a barecommand-error.validaterefuses an oplog holding an op that needs a newer engine version than itsengine-versionpin, withunsupported-on-engine-version(exit 2), asmigrateandsnapshotdo. A log whose pin was lowered by hand, as in a resolved header merge conflict, no longer passes the CI gate.op sealandidentifyrefuse the same log with the sameunsupported-on-engine-version(exit 2):op sealwrites no Seal over it andidentifyresolves no cut.op infodrops the note that said they did not check the engine-version gate.validate,migrateand the other commands that read the log refuse aCreatePartitioncarrying bothpartition_columnsandpartition_expression, as they already refused it onCreateTableandPartitionTable. Such a sub-partition used to replay, andmigraterendered its key from the expression alone, dropping the columns. ACreateTablecarrying both with nopartition_byis refused too, rather than read as a plain table with its key dropped.op addand the commands that read the log refuse anExtractTablethat moves every live column of a partitioned parent or a partition, withextract-partitioned-source. Such an extract ended the source while a parent’s partitions stayed live, or while a partition’s parent still named it, and the renderedDROPdiverged the database from the log.
Changed
- Every
own-envelopemigratefile now carries an open-transaction check: a head ahead ofBEGIN;and a recheck after theON_ERROR_STOPguard. This enforces theowncontract, that the file owns its transaction. Applied inside a transaction that is already open (psql --single-transaction, a runner’s own transaction, psqlAUTOCOMMIT off, a single-Sync batch), the file refuses by name,own-envelope-transaction-open(SQLSTATE25001), before any DDL. That includes cells that happened to work before, such as a lone unsegmented file underpsql --single-transaction, and cells that applied but lost the file’s atomicity at exit 0. The check needsUSAGEonplpgsql. Everyownfile’s bytes change, so a checksum recorded over an earlierownfile (Flyway’s, Liquibase’s) no longer matches a re-emit. - PostgreSQL 18 is a supported engine version, like 15, 16 and 17, at the
PostgreSQL 17 syntax level: every verb writes and reads the SQL it does at
17. The PostgreSQL 18 syntax refuses
syntax-not-available(exit 2) at every engine version:AddColumn generated_kind=virtual,enforced=falseonAddForeignKeyandAddCheck,period=trueonAddForeignKey,without_overlaps=trueonAddPrimaryKey,AddUniqueandChangePrimaryKey, and theSetConstraintEnforcedandSetConstraintInheritoperators.op addandop reviserefuse it, andvalidate,op seal,identify,snapshotandmigraterefuse a log that holds it, anddriftrefuses to append to one.op infomarks these values and operators as not available. migraterefuses a forward window whose reference record is superseded by a laterreplaceof its table while a step between them reads the table (dml-superseded-in-window, exit 2). Before, such a window exited 0 and the dependent step ran on rows the pruned record would have filled, so anAddNotNullcould fail at apply.data captureanddriftwarn with the same code when the floor they mint makes such a window, thesnapshotcapture hint names the split and the version, andop sealrefuses a range that would seal the pair.migraterenders aPartitionTableof several levels as one nestedPARTITION OFtree behind aLOCK TABLE ... ACCESS EXCLUSIVEand a guard that refuses, before any change, rows the declared tree does not route or that aCHECKof the window rejects. ADepartitionTablerebuild opens with the same lock, and the target of one must be the root: a leaf or middle refusesdepartition-target-is-partition. Its warnings now walk every descendant, naming each partition-owned index, constraint, policy and trigger (an inherited copy is not named), each partition’s stamped DEFAULT, NOT NULL and comment, and each kept serial sequence the rebuild drops; they say the down across it refuses for now. The generated-column recompute warning says the same.ExtractTableof a column that a partition-owned element or a middle’s partition key names refuses, as it did for the source’s own.op infoshows, for each rule, what retrofit does to an op that breaks it at two moments: OP ADD (op addand the other commands that write an op) and READING THE LOG (migrate,snapshot,validate,op seal,identify), eachrefusesorlets through.--as jsonkeepswhenand addsenforcement: {"append": ..., "replay": ...}, derived from it.op infosays READING THE LOGrefusesforpartition_cols_xor_expronCreateTable,CreatePartitionandPartitionTable, and forview-replace-supersetonReplaceView: the commands that read the log refuse an op that breaks them, whichop infosaid they let through.--as jsonreports these rules withwhen: append+replay,enforcement.replay: refusesandbinding: binding, no longerauthor-time-convenience.MergeTable’s rules listMergeShapeonsources: the refusal it already raised, when a source’s columns differ frominto’s or a source has a nextval default or no live column, now has a declaration.- Advisory messages end without a period. The second sentence of
data-motion-partition-not-pg-declarativeand ofmove-column-down-refused, the advice, moved to the hint. Amigrate --downscript’sreverse-unrecordedline ends; the database sits at <opid>. The five witness warnings on a record the down reverts state their cause instead of repeating their code, anddata-motion-source-consumed,dml-body-partition-dependent,data-only-single-statement-autocommitand the two drift-skip warnings are reworded in plain terms. data-loss-on-applymessages end without a period too: the reasonsop add,migrateandmigrate --downbuild for a drop, a cast, a join, an unjoin, a matview replace and a lookup extract or inline. Drift’s extract-recognition diagnostics read “an ExtractTable”.- The Advisory codes page shows every form of a message that varies with a field, each led by the value that selects it, and follows a message that is only a placeholder with its detail and hint.
Removed
migrate --allow-unreached-dmland itsdml-ahead-of-windowwarning. A data record anchored after--tois outside the window and is now omitted without a refusal, a warning or a flag, as a reference or seed record past--toalready was. Promoting an environment that sits behind the tail no longer needs the flag; drop it from scripts, where it is now an unknown flag.dml-anchor-unresolvednow means only an anchor the oplog does not hold, fixed withretrofit data repin.LayerSeparationandNotSealedno longer appear in any operator’s rules: retrofit never checked either. Layer separation is a property of the oplog’s slots (docs/concepts.md, “The oplog”); the sealed region is guarded bysealed-region-write.
Known issues
- Ending a column that a later
RunRawScriptbody names is not flagged:op removeorop reviseof the column, or a merge whose other side removed it, writes the log, andvalidateaccepts it.migratethen emits the script against a table without the column, so the migration fails whole at apply. To catch it before landing, have CI apply the migration to a scratch database (retrofit migrate --from root --to last, or from the latest Seal). identifyon a dump that matches no cut can take minutes, where a dump that matches answers in seconds. In one measured case, a history of about 120 operations, a dump matching no cut took 137 s against 0.57 s for one that matched.driftover a database that skipped a migration the log records as a drop, as an out-of-order deploy or a migration-folder tool running an older pending migration does, re-adds the dropped column as anAddColumn, with no question and no note. Review a drift plan’s re-adds against the log before accepting them. A fix is planned after v0.10.0.- A released
transitionorcleansedata record is protected only by its position in the log: an edit to one behind a Seal passesvalidate, andmigrateemits the edited bytes. Nor is a record the release shipped told apart from a branch’s backfill integrated behind it, which a database already past that point never runs. - Before 1.0 the oplog may gain operators and header keys. A build refuses
a log with a newer
retrofit-oplog-formator a key it does not know, so CI and every developer upgrade together. Planned after this release:op record;Archive,UnarchiveandPurge;AnnotateandRetract; erasing a released record; renaming a schema; possibly a marker for experimental operators.