retrofit
0.15.0-wip.89
retrofit.sh

These docs are for 0.15.0-wip.89, a WIP, and may describe what no release does yet. The newest release is 0.14.0: this page for 0.14.0.

Data files

The oplog records schema changes. The rows a schema’s tables hold, as INSERT, UPDATE, DELETE and MERGE statements, are kept beside it in data files: runnable .sql files, one table each, under the schema’s data_root (data by default). You commit them with the oplog.

import writes a data file for each row statement it finds in a source, and data capture, drift and data scaffold write them too. Work with row data shows how to write one by hand.

How a data file follows the schema

Each data file names an op: the point in the oplog its statements were written against. When snapshot or migrate emits the file, it brings the table and column names up to date with any rename since then, and refuses a statement it cannot make true. The file on disk never changes. Stale data references and repair has each case.

What a data file is for

Each data file has a purpose, which decides what the emit commands do with its rows:

PurposeHoldsWhat snapshot and migrate do
refThe rows of a table declared reference data (schema ref-tables add), such as a list of countriesApply them on every run
seedA named dataset: fixtures, or one environment’s rowsApply them when --label names the dataset
transitionOne-time statements tied to a schema change, such as a backfillmigrate runs them right after that change
cleanseThe same, for statements that must not run twicemigrate runs them once, even when the migration is applied again
unclassifiedA file with no purpose setmigrate refuses it; snapshot leaves it out

Experimental purposes

transition and cleanse are experimental, and a project turns them on in retrofit.toml (Experimental purposes). A Seal’s digest covers the oplog but not the data files, so an edit to a released transition goes undetected. A backfill recorded in the oplog instead, as a DO block in a RunRawScript op, is covered.

The data file is the file format in full.

Stale data references and repair

A preserved data record is captured against the schema as it stood at its anchor: the op it is filed under. The schema keeps moving after that. By the time the record is emitted, a column it names may have been renamed, its table may have been renamed or dropped, or a column may have gained a constraint the stored rows cannot satisfy.

That is decided per record, at emit time, and the stored file is never rewritten. There are three outcomes:

  • The table is gone. The record is omitted and reported as excluded, cause table-dropped. Its rows have nowhere to land, so there is nothing to emit and nothing to fix.
  • The record can be made true. The emitted text is rewritten through the oplog: a renamed table or column is emitted under its current name, a column dropped from an INSERT column list (or an UPDATE ... SET left-hand side) is removed along with its value, and an enum label renamed by RenameEnumValue is emitted under its new spelling where the value is written straight into a column of that enum: an INSERT value, or the right-hand side of UPDATE ... SET or ON CONFLICT ... DO UPDATE SET, bare or cast to the enum. A label reached only through an expression (a WHERE predicate, INSERT ... SELECT) or held in an array of the enum is left as written, and a COPY body is reported as a type change rather than rewritten. The emitted body has a comment naming each repair, one courtesy line per repaired table lands on stderr, and a machine-readable record is in the JSON stream. Your data file on disk is untouched.
  • The record cannot be made true. A column the body reads was dropped, a new NOT NULL column has no default, a cast is lossy, two captures of one table have forked. The run exits non-zero with stale-dml-reference, naming every refused record and its reason; the hint names the recourse: re-capture the table, or for a forked chain retire one file.

Repair follows identity, not names. Two references name the same thing when they resolve to the same object in the schema at the record’s anchor, so a table renamed twice still repairs, and a column dropped and re-added under the same name is refused rather than silently reused.

A name retrofit does not model (a function, a sequence, an unmanaged table) is left exactly as written. --verbose lists them per body so you can see what was passed through untouched.

Capturing from a corpus label

Rows captured under a cohort label (data capture <source> --label <l>) land on that label’s seed[<l>] chain. Once the table is declared reference data (schema ref-tables add <t>), data capture --from-label <l> [--table <t>] promotes them: the corpus is the source, so no file operand is given and no database is read. The verb folds the label’s chain for each selected table at the current head and captures the result on the ref band as one literal INSERT with an explicit column list, each value copied from the stored records. Every selected table is checked before anything is written: an undeclared named table, a chain that does not fold offline, or an empty fold refuses the whole run. The seed chain itself is left as it is.

Which point a record is rendered against

A record is rendered against two points in the oplog: its anchor, and the point the verb is emitting for. Only the second one moves, which is why one stored file can emit verbatim, repaired, or refused across runs without changing on disk.

  • migrate has a window, --from to --to, and emits the records whose anchor falls inside it, each rendered against the window’s end. A record anchored before --from is not emitted at all; a dml-behind-window warning names it so nothing is dropped silently. Widen the window if you meant to include it. A record anchored after --to is outside the version the window renders and is omitted without a warning. An anchor the oplog does not contain cannot be placed in any window, and the run stops with dml-anchor-unresolved.
  • snapshot has no window. --at <opid> is the one point it renders for (the tail when omitted), and every opted-in record is rendered against it. --at the record’s own anchor therefore emits the stored body byte-for-byte; --at a later op emits it repaired, or refuses, as the ops in between require.
  • On snapshot, a record whose anchor precedes the history you have (an oplog that starts after it, or an op it never contained) has no schema to be resolved against. Its names are left exactly as written and the body is emitted verbatim, exit 0; --verbose lists them as unmanaged references, the same way an unmodelled name is listed.

Attesting a stale reference

--allow-stale-dml on migrate and on snapshot downgrades every refusal in the run to a warning: repairs are still applied, the body is emitted under a -- WARNING: stale reference comment naming the reason, and the run exits 0. It is a deliberate override, not a repair: you are asserting you know the emitted SQL is right anyway.

One case it does not cover: two data files claiming the same table under one label set through a rename (a forked chain). There is no emission that is correct for both, so the run refuses with or without the flag. Retire one of the files and re-run.

A record routed into a product’s floor that the routing cannot place is attested but never emitted. A product that takes its floor from its source (state: what the fold decided) takes its records from the source’s, and a routed record whose source body cannot be rewritten into the product keeps that body, so the flag downgrades its refusal and leaves the record out. The warning says so (attested via --allow-stale-dml, but not emitted). In migrate, the flag does not lift a key-uncovered refusal on a replace floor the record leaves empty. A capture of the product at HEAD gives it rows of its own.

Before 1.0, any release may change behavior; the release notes list what changed.

Published .