Data files
The oplog records schema changes. The rows a schema’s tables hold, as
INSERT, UPDATE, DELETE and MERGE statements, are kept beside
it in data files: runnable .sql files, one table each, under the
schema’s data_root (data by default). You commit them with the
oplog.
import writes a data file for each row statement it finds in a
source, and data capture, drift and data scaffold write them too.
Work with row data shows how to write one by hand.
How a data file follows the schema
Each data file names an op: the point in the oplog its statements were
written against. When snapshot or migrate emits the file, it brings
the table and column names up to date with any rename since then, and
refuses a statement it cannot make true. The file on disk never
changes. Stale data references and
repair has each case.
What a data file is for
Each data file has a purpose, which decides what the emit commands do with its rows:
| Purpose | Holds | What snapshot and migrate do |
|---|---|---|
ref | The rows of a table declared reference data (schema ref-tables add), such as a list of countries | Apply them on every run |
seed | A named dataset: fixtures, or one environment’s rows | Apply them when --label names the dataset |
transition | One-time statements tied to a schema change, such as a backfill | migrate runs them right after that change |
cleanse | The same, for statements that must not run twice | migrate runs them once, even when the migration is applied again |
unclassified | A file with no purpose set | migrate refuses it; snapshot leaves it out |
Experimental purposes
transition and cleanse are experimental, and a project turns them
on in retrofit.toml (Experimental
purposes). A Seal’s
digest covers the oplog but not the data files, so an edit to a
released transition goes undetected. A backfill recorded in the oplog
instead, as a DO block in a RunRawScript op, is covered.
The data file is the file format in full.
Stale data references and repair
A preserved data record is captured against the schema as it stood at its anchor: the op it is filed under. The schema keeps moving after that. By the time the record is emitted, a column it names may have been renamed, its table may have been renamed or dropped, or a column may have gained a constraint the stored rows cannot satisfy.
That is decided per record, at emit time, and the stored file is never rewritten. There are three outcomes:
- The table is gone. The record is omitted and reported as excluded,
cause
table-dropped. Its rows have nowhere to land, so there is nothing to emit and nothing to fix. - The record can be made true. The emitted text is rewritten
through the oplog: a renamed table or column is emitted under its
current name, a column dropped from an
INSERTcolumn list (or anUPDATE ... SETleft-hand side) is removed along with its value, and an enum label renamed byRenameEnumValueis emitted under its new spelling where the value is written straight into a column of that enum: anINSERTvalue, or the right-hand side ofUPDATE ... SETorON CONFLICT ... DO UPDATE SET, bare or cast to the enum. A label reached only through an expression (aWHEREpredicate,INSERT ... SELECT) or held in an array of the enum is left as written, and aCOPYbody is reported as a type change rather than rewritten. The emitted body has a comment naming each repair, one courtesy line per repaired table lands on stderr, and a machine-readable record is in the JSON stream. Your data file on disk is untouched. - The record cannot be made true. A column the body reads was
dropped, a new
NOT NULLcolumn has no default, a cast is lossy, two captures of one table have forked. The run exits non-zero withstale-dml-reference, naming every refused record and its reason; the hint names the recourse: re-capture the table, or for a forked chain retire one file.
Repair follows identity, not names. Two references name the same thing when they resolve to the same object in the schema at the record’s anchor, so a table renamed twice still repairs, and a column dropped and re-added under the same name is refused rather than silently reused.
A name retrofit does not model (a function, a sequence, an unmanaged
table) is left exactly as written. --verbose lists them per body so
you can see what was passed through untouched.
Capturing from a corpus label
Rows captured under a cohort label (data capture <source> --label <l>)
land on that label’s seed[<l>] chain. Once the table is declared
reference data (schema ref-tables add <t>), data capture --from-label <l> [--table <t>] promotes them: the corpus is the source, so no file
operand is given and no database is read. The verb folds the label’s chain
for each selected table at the current head and captures the result on the
ref band as one literal INSERT with an explicit column list, each value
copied from the stored records. Every selected table is checked before
anything is written: an undeclared named table, a chain that does not
fold offline, or an empty fold refuses the whole run. The seed chain itself
is left as it is.
Which point a record is rendered against
A record is rendered against two points in the oplog: its anchor, and the point the verb is emitting for. Only the second one moves, which is why one stored file can emit verbatim, repaired, or refused across runs without changing on disk.
migratehas a window,--fromto--to, and emits the records whose anchor falls inside it, each rendered against the window’s end. A record anchored before--fromis not emitted at all; adml-behind-windowwarning names it so nothing is dropped silently. Widen the window if you meant to include it. A record anchored after--tois outside the version the window renders and is omitted without a warning. An anchor the oplog does not contain cannot be placed in any window, and the run stops withdml-anchor-unresolved.snapshothas no window.--at <opid>is the one point it renders for (the tail when omitted), and every opted-in record is rendered against it.--atthe record’s own anchor therefore emits the stored body byte-for-byte;--ata later op emits it repaired, or refuses, as the ops in between require.- On
snapshot, a record whose anchor precedes the history you have (an oplog that starts after it, or an op it never contained) has no schema to be resolved against. Its names are left exactly as written and the body is emitted verbatim, exit 0;--verboselists them as unmanaged references, the same way an unmodelled name is listed.
Attesting a stale reference
--allow-stale-dml on migrate and on snapshot downgrades every
refusal in the run to a warning: repairs are still applied, the body is
emitted under a -- WARNING: stale reference comment naming the reason,
and the run exits 0. It is a deliberate override, not a repair: you are
asserting you know the emitted SQL is right anyway.
One case it does not cover: two data files claiming the same table under one label set through a rename (a forked chain). There is no emission that is correct for both, so the run refuses with or without the flag. Retire one of the files and re-run.