retrofit migrate
Emit a one-way migration script between two cuts
Usage:
retrofit migrate [flags]
Flags:
--allow-external-atomicity [apply context] attest that the apply context owns transaction atomicity: the no-wrapper advisory is not raised where a non-own envelope suppressed the wrapper
--allow-reading-replace-body downgrade the replace-body-reads-relation refusal to a warning; emit the body verbatim under a WARNING comment, and render its table's prologue as a whole-table clear
--allow-stale-dml downgrade every stale-DML-reference refusal to a warning; emit the body under a stale-reference WARNING comment. It does NOT cover chain-collision, which is not attestable: two live chains for one element is a fork to resolve by retiring one, and exit 2 stands
--allow-unclassified-dml downgrade every unclassified-DML refusal to a warning; emit each unclassified body as a trailing unclassified-DML WARNING block
--allow-unnamed-label stringArray attest that the dataset carrying this exact label is deliberately left unnamed in this run (repeatable; byte-exact, never trimmed); every other unnamed dataset still warns
--allow-unreached-dml downgrade the dml-anchor-unresolved refusal for a record anchored AFTER --to to a warning; the record is omitted from this script, never emitted out of position. It does NOT cover an anchor the oplog does not hold, which is not attestable: that file needs a re-pin (retrofit data repin), and exit 2 stands
--allow-unscreened-dml attest set-aside DML whose body carries transaction control or psql meta-command lines (or which retrofit cannot parse): emit it anyway and report the window's segment count as a lower bound rather than an exact number
--at string requires --down: the cut the database is at, an opid prefix or anchor (default: last); --to beside it rolls back the slice (--from, --to] and keeps the ops after it
--data-root data_root root that overrides the schema's data_root
--down emit the reverse of the migration window: each touched element's net transition back to --from, rendered by the forward renderer
--envelope own transactional context the emitted file is applied in: own (retrofit wraps it in one transaction; apply outside any open transaction) | caller (you open the transaction; retrofit emits no wrapper) | independent (each statement applies in its own implicit transaction); default: own
--extra-seed-path seed_paths root that appends to the declared seed_paths
--from root from-opid prefix or anchor (root/`last`/`seal`, optional `^`/`~N`); default: latest Seal, or `root` when no Seal exists
--from-dump string identify the cut the database is at from a pg_dump --schema-only file; it fills --from (forward) or --at (under --down)
-h, --help help for migrate
--help-all show extended help (vocabulary, semantics, caveats)
--into-schema string emit a copy script into <schema>; absence = in-place
--label stringArray apply the dataset carrying this exact label (repeatable; argv order is layer order; byte-exact, never trimmed; never cuts ref data)
--lock-timeout string bound each lock wait (per lock acquisition, row locks included): emit SET LOCAL lock_timeout = '<duration>' at the head of every transaction the file opens (a positive integer with ms, s, min or h, e.g. 2s; off = unset); overrides [project] lock_timeout; refused where the file has no transaction to scope it into
--no-header suppress engine-version banner
--schema <name> registered schema <name> (resolves via the chain when omitted)
--seed-path seed_paths root that replaces the declared seed_paths
--to root to-opid prefix or anchor (root/`last`/`seal`, optional `^`/`~N`); default: last
--using string one-shot emit-target override (format: <engine>:<major>)
Global Flags:
--as string output mode: text | json (default "text")
-C, --chdir string treat <dir> as the working directory for the entire run
--config string retrofit.toml path (overrides discovery; also accepts $RETROFIT_CONFIG)
-f, --file string oplog file path; selects a registered [[schema]] path (overrides discovery)
--no-consolidate render every advisory occurrence instead of collapsing repeats
-q, --quiet count suppress the advisory tier on stderr; stack (-qq) to drop it whole (errors and results still print)
-v, --verbose reveal info-tier advisories on stderr
Docs: https://docs.retrofit.sh/current/commands/retrofit-migrate/retrofit migrate --help-all adds:
Extended help:
Finding opids for --from / --to:
Default `retrofit op show` emits each op in on-disk record
grammar with the opid as the first field:
AddColumn(988edcb612ab, prev=7a6b1234abcd, in=…, name="email", …)
^^^^^^^^^ ^^^^^^^^^^^^
operator opid (12 hex)
Pass any unique prefix (>=4 chars) to --from / --to; both accept opid
prefixes and the canonical anchors `root` / `last` / `seal` with an
optional `^` / `~N` decrement (e.g. `seal^`, `last~2`). An explicit
--from anchor that steps past the oldest element of its line (e.g.
`seal^` on a one-seal oplog, `last~9` on a short log) clamps DOWN to
`root`: the floor of a window is the oldest cut, never a refusal.
(`root^` / `root~N` still refuse `ref-out-of-range`: root is already
the floor, so an under-decrement of it is a typo, not a window.) A
--to (ceiling) or `snapshot --at` anchor stays strict and refuses
`ref-out-of-range` on the same over-decrement.
Useful narrowing flags on `op show`:
--tail N keep the last N matching ops
--head N keep the first N matching ops
--filter key=value scalar equality predicate (repeat for AND)
--touching <name> match ops referencing this element
--framing=add render as runnable `retrofit op add ...` lines
--as=json NDJSON record per op (full opid per record)
Default --from is the most recent Seal, or `root` when the oplog has no
Seal yet; default --to is `last`. Together the defaults render the
unpublished delta (last-seal-or-root, last]. With no Seal that is the
whole oplog as a forward CREATE script, identical to --from=root. An
empty but well-formed window (steady state after a seal, --from=seal
--to=seal, or an empty oplog) is a disclosed no-op (exit 0, an empty
migration document plus a stderr advisory), not a refusal; refusals are
reserved for malformed ranges (range-inverted / opid-not-found /
fork-in-range). --from-dump <file> resolves the cut the database is at from a
pg_dump capture, filling and excluding one slot: --from forward, --at under --down.
--down: reverse migration
Emits the reverse of the window: for each element the window touched,
the net transition from the tip back to --from, as catalog operations
the forward renderer renders. An element live at both cuts is matched
by identity (RenameColumn and RenameTable rename back, a cast casts
back); an element the window created is dropped (a standalone
AddColumn reverses as a DROP COLUMN); an element it dropped comes back
empty; a structural op inverts as its paired operator, which keeps the
rows. An op that only records a proven invariant (ValidateConstraint)
contributes no DDL (note code reverse-noop). The header's report names
every step that destroys rows (data-loss-on-apply-render-chosen, the
statement marked `-- destructive:`), every rule it re-imposes
(reverse-guard) and what it does not bring back (reverse-irreversible).
A window with no inverse refuses migrate-non-reversible, naming each
block and its way back.
--envelope: the transactional context you apply the file in
The envelope is an ASSERTION YOU MAKE about your runner. migrate cannot
see how the file is sent, so it emits according to what you declare:
own (default) retrofit wraps the file in one transaction; apply it
outside any open transaction of your own
caller you open the transaction; retrofit emits no wrapper
independent each statement applies in its own implicit transaction
Resolution is --envelope -> the [project] envelope key -> own. An unknown
value is rejected before the verb runs.
`own` wraps on every path, --down and --into-schema included. Any run
that emits no wrapper carries a `wrapper-suppressed` advisory whose
detail names EVERY input that suppressed it, not just one. Read that
advisory rather than inferring atomicity from the envelope you asked
for. --envelope is registered on
migrate only; snapshot's wrapper is a separate surface keyed on
--include-drop.
Some statements need the enclosing transaction closed around them:
`ALTER TYPE ... ADD VALUE` needs it committed before a later statement can
use the new label, and `DETACH PARTITION ... CONCURRENTLY` cannot run
inside a transaction block at all. Under `own` retrofit emits that
boundary itself and the window emits as segments (see below). Neither
no-wrapper value gets one, and they do NOT behave alike:
caller REFUSES such a window (migrate-tx-incompatible, exit 1).
You declared a transaction of yours open around the file,
so a bare mid-file COMMIT; would end YOUR transaction
early, leaving whatever already ran committed
independent EMITS, with a migrate-segmented-non-atomic advisory on
stderr (exit 0). No boundary is needed: per-statement
transactions already satisfy the requirement, so there
is nothing to attest to
A boundary that resolves without costing atomicity, a trailing commit
boundary with nothing after it to read the effect, neither refuses nor
claims segmentation; it emits a per-operator boundary-requirement-*
advisory instead, because the obligation is now your apply context's.
Segmented scripts, and what they report:
A window emits as N independently committing segments, rather than one
all-or-nothing script, when one of the boundary-needing operators named
above is in it. Ordinary DDL never segments, however much of it there
is, and neither does screened data on its own. (Data you admit past the
screen with --allow-unscreened-dml is the exception: a body carrying its
own transaction control can segment a window with no such operator in
it at all, which is why that attestation degrades the reported count to
a lower bound.)
Whether a boundary costs atomicity depends on which positions hold a
statement on each side of it, and two kinds of position can:
operators the boundary-needing statements named above
data records a .data.<format> record occupies a position too. Seed records
(named on --label) are emitted after ALL DDL, so they land past
every operator; a root-anchored record lands ahead of
every operator; an operator-anchored record shares that
operator's position
The two boundary-needing kinds differ here. `ALTER TYPE ... ADD VALUE`
needs only a COMMIT before a later reader, so a trailing one, with
nothing after it to read the effect, separates no work and costs no
atomicity. `DETACH PARTITION ... CONCURRENTLY` must run outside every
transaction, which costs atomicity wherever it sits, last position
included. Adding a trailing seed block puts something after the first
kind: a window whose last operator needs a commit boundary segments once
the seed occupies the position past it: the DDL commits, then the data
applies as its own unit. Narrowing --from/--to cannot separate that
pair, since the seed band is emitted after every operator in any window.
A segmented script, and any script bounded by --lock-timeout, carries
an ON_ERROR_STOP guard as its first body line, so a failure (a lock
timeout give-up included) halts instead of running on into later
segments or statements.
That costs atomicity. Under `own`, where retrofit places the boundaries
and the runner opens none, such a window emits as segments and reports
the cost as a migrate-segmented-non-atomic advisory on stderr (exit 0)
naming each boundary and what a failure in a segment leaves durable.
The all-or-nothing it loses is the window's, not the envelope's:
`ALTER TYPE ... ADD VALUE` followed by its use cannot be one transaction under
any envelope, so no flag is asked of you:
--allow-non-atomic-segments is retired, and passing it refuses.
--allow-external-atomicity does not answer this advisory: it attests the
apply context, and the lost atomicity is the window's. To keep the
window atomic instead, narrow --from/--to so the responsible operators
fall outside it.
migrate-tx-incompatible covers more than one situation, so the refusal
carries a stable class discriminator you can branch on (`class:` on the
line after the message in text mode, `discriminator` in the JSON error
envelope). Read the class before choosing a remedy; no flag answers
either:
tx-incompatible-unattested
the window needs a boundary retrofit does not own: raised under
caller. The remedies are narrowing --from/--to, --envelope=own,
or splitting the migration into separate files
tx-incompatible-unsatisfiable-edge
there is NO legal segmentation: one position demands the
transaction closed while another demands that same transaction
stay open, so no envelope or flag can help. The only remedy is to
SPLIT THE WINDOW so the conflicting operators land in separate
migrations
Both classes are exit 1. Under own and independent the unattested class
never fires and unsatisfiable-edge reports directly; under caller the
unattested class fires first.
A spliced data body carrying its own COMMIT;/BEGIN;/ROLLBACK;, or a
statement PostgreSQL bars from a transaction block, would open a unit
the count never saw, so every body is parsed and screened first and such
a body is refused (migrate-dml-unscreened). --allow-unscreened-dml emits
anyway and degrades the reported count to "at least N segments".
Apply a segmented script with a runner that opens no transaction of its
own (plain `psql -f`). Under `psql --single-transaction` the
mid-file `BEGIN;` only warns, but the mid-file `COMMIT;` ends the
runner's transaction early: the apply succeeds and psql exits 0 over a
schema that is no longer all-or-nothing, with no error to catch. An
unsegmented `own` file is not exposed: its one `COMMIT;` closes the
whole body. Because segments commit independently, a failure in segment
k leaves 1..k-1 durable and re-emitting is byte-identical, so recovery
means inspecting which segments applied and hand-authoring the
remainder.
Apply with `psql -X -v ON_ERROR_STOP=1 -f`, under any envelope. psql
reads its startup file (~/.psqlrc) unless -X is given, and
`\set ON_ERROR_ROLLBACK on` there savepoints every statement inside a
transaction: a failing statement (a guard's RAISE included) no longer
aborts the file's transaction, the rest runs, COMMIT; commits, and psql
exits 0 over a partial apply. A file with no ON_ERROR_STOP guard of its
own (one span, no --lock-timeout) is exposed under own and under your
own BEGIN; \i file.sql COMMIT; wrapper. -X skips the startup file, and
-v ON_ERROR_STOP=1 halts at the first error with exit 3.
Preserved DML (the data corpus):
An import that contributes new preserved records mints one addressed data
file per table (<schema>.<table>.<YYYYMMDDHHMM>[.append][.p<K>].data.sql) under the schema's declared [[schema]] data_root (default: the oplog directory).
An exact re-ingest is deduplicated against the corpus and mints no file.
migrate recursively discovers every *.data.<format> (and legacy *.dml.sql) under that corpus root and
routes its records by purpose:
ref base corpus: inline at its anchor; always applied;
never label-filtered
transition inline, right after its anchor-op's DDL
cleanse inline, wrapped in a run-once marker
seed base corpus: opt-in by naming the cohort on --label, as one
trailing block after all DDL. --label is the WHOLE opt-in and
is repeatable, lowest layer first; a seed chain no --label
names is not applied, and is counted in the exclusion report
unclassified refused (dml-unclassified, exit 2) unless
--allow-unclassified-dml downgrades it to a warning
Inline records apply only when their anchor is in the migration window;
a behind-window anchor is omitted with an advisory; an anchor ahead of
--to refuses (dml-anchor-unresolved) unless --allow-unreached-dml
omits the record under a warning; an anchor the oplog does not hold
refuses, and the recourse is retrofit data repin. In the emitted text, the
body follows table and column renames and drops removable write-only
references to dropped columns; dropped-table records are excluded. migrate
refuses a reference it cannot repair (stale-dml-reference, exit 2) unless
--allow-stale-dml downgrades it to a warning. A declared data_root that
does not exist refuses (data-path-missing); an absent undeclared default
or an empty corpus directory is the normal no-corpus case.