retrofit
0.15.0-wip.89
retrofit.sh

These docs are for 0.15.0-wip.89, a WIP, and may describe what no release does yet. The newest release is 0.14.0: this page for 0.14.0.

retrofit data: the data corpus

The data verb group works on the data corpus: every addressed *.data.<format> data file under a schema’s declared data_root (written as .data.sql). Data files hold row data (reference tables, fixtures, migration backfills) beside the DDL history the oplog records, and the emit verbs (snapshot, migrate) fold them into a data band at emit time. Each verb’s flags are on its own page in the Command reference, which the table links.

VerbWhat it does
data listShow the fold the emit verbs would apply. Read-only.
data captureCapture observed rows from an INSERT-style dump.
data retireWithdraw one table’s data chain at the current head.
data scaffoldMint a data file for a table holding one inert placeholder record to fill.
data checkValidate the corpus against the data-file grammar.
data witnessBind reference records’ bytes to the oplog.
data repinRe-point one data file’s anchor-op= at an opid you name.
data reanchorRe-anchor records whose anchor was revised, removed or sealed past onto the op that now holds their place.
schema ref-tablesRead or edit which tables hold reference data.

Synopsis

From retrofit data --help:

Usage:
  retrofit data
  retrofit data [command]

Options

From retrofit data --help:

Flags:
  -h, --help       help for data
      --help-all   show extended help (vocabulary, semantics, caveats)

The global flags are in the Output contract.

retrofit data list

data list is the consumer-side dry run of the data band. It answers “what would snapshot / migrate actually apply?” over the same fold those verbs read, and it emits no SQL and writes nothing.

retrofit data list                                 # the reference fold, then the label inventory
retrofit data list --label env:dev                 # the fold for one dataset
retrofit data list --label base --label env:dev    # two layers, in that order
retrofit data list --label env:dev --at op48       # the fold at an earlier cut
retrofit data list --schema sales --label env:dev  # a non-default schema

Work with row data walks a run with no --label and a run that names two datasets.

The fields on a line

data list --help-all defines each token of an item line. Beside it:

  • Items are one line per table, in bytewise table order. An item line is the table, its state (below), then floor=, witness=, removed=, appends=, excluded= and s3= in that order; witness= and removed= follow a floor only.
  • A bare run (no --label) folds the reference band only, counts each labeled chain’s records under not-selected, and ends with one label <label> files=N line per label, in bytewise label order.
  • witness= reads witnessed <opid> when a witness governs the floor record and its bytes still match, witness-stale when they no longer do, and unwitnessed when no witness governs it. Witnesses govern reference records only, so every seed floor reads unwitnessed.
  • --label is ordered: each occurrence is one layer, and a later layer’s replace floor supersedes the layers below it for that table. A file holding several named labels folds in each of those layers and is emitted once, at the highest of them.
  • When the run repairs a record for a schema change since its anchor (a rename, say), a notice: <table>: … line on stderr precedes the items.
  • Under --as=json, appends[], excluded[] and s3[] name each entry the text line counts. An s3[] entry’s lower_label is the layer of the floor it is anchored below, as floor= spells it (ref for the reference band).

s3=1 below says env:dev holds one contribution to public.accounts anchored before base’s floor at op61; --as=json gives its anchor:

$ retrofit data list --label base --label env:dev
public.accounts  occupied  floor=base/replace/data@op61  witness=unwitnessed  appends=1  excluded=0  s3=1

state: what the fold decided

Each item has one state literal:

stateMeaning
occupiedA floor selected, and rows survive to the emitted band.
emptyA floor selected, and it declares the table empty (body=none).
composedNo single floor; the band is composed from appends alone.
unspokenThe corpus holds records for this table, but this run’s labels named none of them.
noneNo record in the corpus addresses this table at all.
droppedThe table was dropped from the schema, or ended by a JoinTable, SplitTable, MergeTable, InlineTable, UnjoinTable or an ExtractTable that consumes its source; its records can no longer apply.

A dropped table’s records are listed in excluded[] under table-dropped. A table one of those ops ended takes that op’s own cause instead (table-joined, table-split, table-merged, table-inlined, table-unjoined or table-extracted). Nothing is owed on the source’s records: they stay as history, and a witnessed one stays verifiable.

Some products take their floor from the source: a CopyTable clone, each ExtractTable child (one that copies the source’s primary key, and a lookup) and each UnjoinTable child the op gives a key. The floor is the source’s records as the run folds them just before the op, routed into the product, so a fresh build holds the rows a migrated database holds, and nothing is owed on the product. A capture of the product starts its own floor after the op. Every other product is captured on its own: a JoinTable, SplitTable, MergeTable or InlineTable product, and an UnjoinTable child with no key. For a declared one, op add advises the capture (data-motion-product-capture-owed), and op seal refuses reference-rows-uncaptured until the product has a reference record anchored at or after the op.

A seed record on a table inside its reference span is dormant: while the table is declared at the render point, data list, snapshot and migrate never fold its seed records, stack them over the reference rows, or apply them alone, whether or not the table holds reference records yet. A run that names the chain’s label lists them in excluded[] under reference-data, and a table whose only named records are dormant reads unspoken, never none. The seed chain is kept and never reclassified.

The emit verbs warn once per dormant chain (apply-risk-data-dropped, cause reference-data, at every verbosity), naming the chain’s recourse by the table’s name at HEAD:

  • For a one-label chain, retrofit data capture --from-label <label> --table <table> promotes the fixture’s rows into the reference chain.
  • For a chain with several labels, which --from-label cannot fold, the warning names a capture of the table from a database instead: retrofit data capture <dump> --table <table>.
  • retrofit data retire --label <label> <schema.table> withdraws a chain that was only a stand-in; for a chain with several labels, the line names every label.
  • When HEAD no longer has the chain’s table (a historical --at inside a span whose table was later dropped), the warning names no command.

A --schema you typed is repeated on the capture line. Outside every span, before the declaration or after schema ref-tables remove, the seed chain is an ordinary dataset again.

Refusals

data list refuses where snapshot’s corpus scan and fold refuse, with the same code, and never prints a partial listing. A declared data_root that does not exist refuses data-path-missing, and a corpus holding a transition or cleanse record the project has not opted into refuses experimental-not-enabled (Experimental purposes). An unreconciled fork between the first op and the cut refuses fork-in-range, as snapshot does; an --at at or before the fork’s shared prev= lists normally. Refusal codes gives each code’s exit status.

No witness finding refuses data list, and neither does a record whose anchor-op= names no live op. Where validate, op seal, snapshot, migrate and data check refuse a covered reference record on its witness, or data check, op seal, snapshot and migrate refuse a record on its anchor, data list lists the record and reports the finding instead:

Finding the other verbs refusedata list reports
data-witness-mismatch: the record’s bytes differ from its witnesswitness-stale
data-witness-missing: no witness in the covering seal’s rangeunwitnessed
witness-out-of-range, witness-digest-unsupported: no witness governsunwitnessed
data-file-missing: a witness names a record no file holdsa stderr line listing the record from the oplog, witnessed <opid>
data-file-anchor-unresolved: a record’s anchor-op= names no live op (migrate refuses it as dml-anchor-unresolved)anchor-unresolved under the table’s excluded=, and one stderr note naming each anchor’s fate and the recourse

A record in the unsealed tail whose witness is stale or absent reads witness-stale or unwitnessed too; there only op seal refuses it. To get the refusal and its code, run retrofit validate or retrofit data check for a witness finding, and retrofit data check for an unresolved anchor.

JSON

Under --as=json each item is a {"type":"item", …} record on stdout, followed by the terminal {"type":"ok"}; a bare run emits one {"type":"labels","present":[…]} record after the items and before {"type":"ok"}. See Output contract for the record shapes.

The -- Data: designation line

snapshot and migrate write one comment line above every run of rows they emit, naming the rows’ band:

-- Data: "shop"."statuses" (reference)
-- Data: "shop"."fixtures" (seed, label qa:golden)

The designation is reference, seed, label <label>, transition, cleanse or unclassified. A multi-label chain’s line names its placing layer’s label: the highest-positioned --label that selects the chain. Dropping that label from argv still ships the rows when another named label selects the chain, and the line then names the other label. A labelled ref record reads reference whatever its labels. A replace block that also holds another label’s rows opens that label’s line above them, inside the block, and its own line again below. data capture drops these lines from the bodies it mints and keeps any other comment, including a hand-written -- Data: loaded by hand.

retrofit data capture

data capture <source> records what an INSERT-style dump says each table’s rows are at the current oplog head. It observes state, so it has no --purpose flag.

retrofit data capture rows.sql --label env:dev
retrofit data capture rows.sql --table countries --table 'curr*'
retrofit data capture rows.sql --schema sales --dry-run

A table whose oplog declaration reads reference_data=true at the run’s cut is captured as ref; otherwise a non-empty --label set captures it as seed with exactly that label set; otherwise the table is listed, not captured, and the run exits 0. There is no producer default. --from-label <label> reads a label’s seed fold instead of a source (Capturing from a corpus label).

Capture reads INSERT statements, including the output of pg_dump --inserts and pg_dump --column-inserts. COPY ... FROM stdin, UPDATE, DELETE and MERGE are not capture input; import takes authored DML.

Every captured table is a full observed row set (disposition=replace) at HEAD, and the table’s key at render sets how that replace replays (The replace prologue). A table with no key the rows write in full is cleared whole-table and reloaded when no other table’s foreign key references it, and refuses key-uncovered when one does. A secondary UNIQUE the arbiter does not cover, or a surrogate key re-minted at capture, can make an upsert collide or replace rather than update at apply. The recourse is chosen at capture time: declare the constraint DEFERRABLE INITIALLY DEFERRED (which removes it from arbiter candidacy; if the key is then a unique index on the same columns, the render refuses replace-no-upsert-form), or, for a table no other table’s foreign key references, capture the table keyless.

A backfill is history, not state: a ref append that fills a new column runs at its anchor, and a later capture of the table supersedes it. When a step between the two reads the table or validates its rows, migrate over that window refuses dml-superseded-in-window (Preserved DML). The capture itself still mints and exits 0, warning with the same code and naming the backfill, the first such step and the recourse, and op seal refuses to seal a range that would make the refusal unsplittable.

Text mode prints one captured … or already-captured … verdict per table. JSON mode emits per-table rehearsal records on a dry run and closes every run with an ok envelope whose data.tables array holds the verdicts. --dry-run writes nothing and exits as the real run would. Rows equal to the chain’s fold are already-captured and mint nothing, with two exceptions:

  • A ref chain at one anchor whose replace floor cannot render at that anchor (migrate would refuse replace-no-upsert-form, or key-uncovered under an inbound foreign key) still mints the fresh floor. The verdict reads supersedes @<anchor> (no-upsert-form at the anchor) or (key-uncovered at the anchor) (supersedes_cause in JSON).
  • When import s.sql has moved a floor to an append (see Re-ingesting into the corpus), capturing s.sql again moves the key back even at identical rows. The capture mints the floor and reports overwrote @<anchor> (append → replace), or supersedes @<anchor> (append → replace) once HEAD has moved (moved_from: "append" in JSON). Only the append’s own source moves it back: another source whose rows equal the append’s, a chain that still holds a floor, and a capture from stdin stay already-captured.

A capture, or a data retire restate, at an anchor whose key holds only an append overwrites that record, every source’s statements, when its rows differ: the verdict reads overwrote @<anchor> and one overwrote: line names each replaced file with its digest. A src=hand record at the key refuses the run data-file-collision.

An unmatched --table pattern is an advisory, not a silent no-op.

A captured ref table that lost a key since its chain is still captured, with a reference-key-lost warning naming the lost keys. A declared table the source observes empty (its only statement an unfired TRUNCATE or WHERE-less DELETE) mints replace body=none and warns reference-table-emptied, --dry-run included; an already-captured re-run, a seed chain and data retire do not warn. drift raises both warnings for the tables it captures.

A <source> that is a stream (-, <(…), a FIFO) delivering zero bytes refuses source-empty. A <source> that is a URL, a non-.sql file or an extension-less regular file refuses kind-mismatch.

Each claimed table’s source band (the bytes of every INSERT statement naming it, summed over the whole source) is bounded by the schema’s data_band_max_bytes (see Create a project). A claimed table over the bound refuses data-band-over-cap before any row is retained or minted, --dry-run included; the envelope names every over-cap table with its bytes, the bound and its statement count. A table --table filters out, or one the run does not claim, is not counted. Recovery: raise data_band_max_bytes for the named schema, or remove the key for the built-in default.

retrofit data retire

data retire <schema.table> withdraws an existing data chain by minting an empty restate floor at HEAD.

retrofit data retire public.countries
retrofit data retire public.accounts --label env:dev
retrofit data retire public.accounts --label base --label env:dev --dry-run

With no label, retire addresses the table’s unlabelled ref chain. Repeated --label flags address the seed chain holding exactly that label set, never a union of matching chains. If no such chain exists, retire refuses no-such-chain and lists the chains that do.

A reference chain ends with its declaration. While the table is declared an unlabelled retire refuses retire-declared-table, naming schema ref-tables remove; once the declaration is removed it mints nothing and reports the table not-managed, exit 0.

Removing a table’s declaration (schema ref-tables remove) ends its reference span, and retrofit stops managing the rows. A fresh build (snapshot at or past the removal, and data list) omits the records anchored inside the ended span, cause undeclared (named under -v). snapshot --at inside the span still renders them, and a migrate window crossing the removal applies them as a delta against the fold at the window’s start, by key, or by whole row where no key identifies one (see A window across a reference declaration’s removal). drift reports the chain not compared, information rather than drift. The records and their witnesses stay as history. Declaring the table again opens a new span; the old span’s records do not resume. A ref record holding rows anchored on or past the removal refuses in every reader as data-file-malformed, class ref-undeclared, and the body=none floor an unlabelled retire minted before the removal is exempt; see data check below.

Text verdicts say retired, already-retired or not-managed; JSON uses the same rehearsal and terminal data.tables shapes as capture.

retrofit data scaffold

data scaffold mints a valid file beneath the resolved schema’s data_root, at the deterministic path for its mint key, holding guidance comments and one inert src=hand record: an INSERT over the table’s live columns with a <column> placeholder per value. A --purpose cleanse file holds a kind=DeleteStmt record instead, DELETE FROM <schema.table> WHERE <predicate>;, whose <predicate> is the placeholder to fill; its guidance comments name the kind= tokens for a record rewritten as an UPDATE:

retrofit data scaffold --table public.backfill --purpose transition --disposition append
retrofit data scaffold --table public.fixture --label env:test --disposition restate

Both --purpose values are experimental: without the project’s opt-in the run refuses with experimental-not-enabled before it writes anything, and with it the run warns experimental-purpose on the file it writes (Experimental purposes).

Scaffold reads the table’s reference-data declaration at its minting cut, as the other producers do. A declared table is minted purpose=ref, and --purpose and --label are ignored for it; a run that passed either raises a dml-ref-table-flag-ignored note naming the flag it ignored. The run reports the minted file the same way for every band; the file’s header line is what says purpose=ref:

dml-preserved: data/app.countries.202609281743.data.sql (declared-axes scaffold, 1 inert record to fill)

--purpose ref refuses with flag-invalid-value, since ref comes from the declaration rather than a flag. With no declared purpose, no label, and no policy match, the file is purpose=unclassified. --disposition takes only append or restate; observed replace data belongs to data capture.

--anchor-op overrides the reconciled head. A witnessed ref key refuses (data-witnessed-key), and an anchor a seal covers refuses (sealed-anchor) for every band but seed. A witness governs the ref chain only, so a record of another band at the witnessed anchor is another key and mints unrefused. A seed record may anchor inside a sealed range, for a regression at the op where the bug lived; the mint raises a seed-anchor-sealed note naming the seal that covers it, since a fresh build and --at render the record there while a database already past that seal never receives it.

A restate of a declared table states the whole table by hand and renders as a replace; the mint raises a reference-restate-authored note (the courtesy band: -q drops it), and data check raises it again over the filled file.

data check is the fill gate: it refuses the file naming every placeholder still present, and migrate and snapshot refuse an unfilled scaffold (dml-body-unfilled) rather than reading it as empty. Work with row data fills a scaffolded record and checks it.

retrofit data check

data check [path...] validates corpus files without emitting SQL or minting anything.

retrofit data check
retrofit data check corpus/fixtures corpus/public.accounts.202609251400.data.sql

The command evaluates every addressed file and reports all findings together. Malformed or conflicting files make the run fail; advisory-only findings such as an unaddressed candidate filename do not change the exit status. A clean text run reports data check: N file(s) checked, no findings; JSON emits a data-check record with checked: N and an empty listed: {}. A run that listed warnings but refused nothing counts them by code instead, as in data check: 1 file(s) checked, no refusals; listed: anchor-shape-mismatch ×1, and the JSON record has "listed":{"anchor-shape-mismatch":1}. The exit is still 0, so a CI step that must stop on a listed warning gates on listed.

data check also routes the floors products take from their source (state). A routed record whose source body cannot be rewritten into the product (a body that does not repair to the op, a statement other than an INSERT, a populated COPY, or a row the op’s guards refuse) is listed as routed-body-unroutable, naming the product, the op, the source record and the reason. It counts in listed, and the exit stays 0; snapshot and migrate refuse the same record as stale-dml-reference. A routed floor a later capture of the product supersedes is not listed.

A record whose anchor-op= names no live op fails the run as data-file-anchor-unresolved. Removing an op ends its id, and a revise that changes an op’s content mints a new id for the new act, so either leaves a record anchored on the old id; a reorder keeps the id, and the anchor survives it. op revise, op remove, op unseal and a single-file reconcile move the records under data_root that their write leaves on a retired anchor (retrofit data reanchor), so the finding names a record that move listed, or one the edit never saw, such as a file a merge brought in. Each detail line names the record’s seq= and anchor, and the anchor’s fate when the oplog still holds it: removed at a line, or replaced by a live opid, the opid a re-pin may name. When a fate is named the hint names retrofit data reanchor first; for one reviewed file, retrofit data repin <file> --anchor-op <opid> re-pins every record in it.

A purpose=ref record holding rows whose anchor-op= resolves to a cut where its table is not declared reference data fails the run as data-file-malformed, class ref-undeclared, naming the file, the table and the anchor. That covers a declaration withdrawn at or before the anchor (schema ref-tables remove; the detail names the op that withdrew it), a table never declared, and a table declared only after the record’s anchor. The header’s table name is compared as the anchor spells it, so a header naming the table by a later name is not declared there either. snapshot, migrate and data list refuse the same file before emitting anything. The recourse is not data repin: remove the file, and to keep its rows declare the table again (schema ref-tables add) and capture them at HEAD (data capture), which mints a record inside the new span. A body=none floor is exempt, since it holds no rows; one anchored after its table’s span ended (the floor an unlabelled retire minted before schema ref-tables remove) is omitted as undeclared by every reader and skipped by data witness. A project with no resolvable oplog yields no such finding.

A purpose=ref restate body=none anchored where its table is declared fails the run as data-file-malformed, class ref-restate-declared, naming the file, the table and the anchor; no producer mints this record (data retire refuses a declared table). The recourse removes the file (rm <file>); to stop managing the table’s rows, then end its span (schema ref-tables remove <table>), which, like every corpus reader, refuses while the file stands. snapshot, migrate and data list refuse the same file. The same file anchored where no span is in force is the exempt floor above; a replace body=none is an observation (see data capture) and a seed restate body=none is data retire’s, and neither is refused.

A filled purpose=ref restate holding rows on a table declared at its anchor passes, with the reference-restate-authored note data scaffold raises (never as data-file-unfilled).

Two files at one key (the same table, band, label set and anchor) fail the run as data-fork, whatever their dispositions: two floors, a floor and an append, or two appends. snapshot, migrate and data list refuse them the same way, under data_root and within each declared seed_paths root (data check reads a seed root when it is named as an operand). A merge that kept both sides, or a hand copy, is the usual cause: a data capture branch and an import branch at one head reach it with no hand editing. The refusal names the key and each file with its disposition and source. The pages of one paginated capture are one record, two bands or two label sets at one anchor are two keys, and a sibling schema’s files under a shared data_root are left to that schema’s own check.

The data-fork hint names one of two recourses:

  • Re-run the delta’s import. For a floor and a delta read from a source file, the hint names removing the delta’s file and re-running its import, which adds the delta as a section of the floor’s record. The re-run repeats the schema the check read (a typed --schema as typed, a schema chosen with -f by its registered name), a typed -C and --config, a typed --data-root (as --seed-path for a labelled delta), and the source relative to the working directory. A delta whose import took --disposition append needs it again.
  • Keep one record. Two floors, a delta with no source to re-run (src=hand, or read from a stream), a floor holding src=hand work or no records, a delta from the floor’s own source, and a re-run that would not land on the floor keep one record by removing the others’ files; a paginated record keeps all its pages. A re-run misses the floor at a key anchored before the oplog’s head (an import anchors at the head), at a floor outside the root it writes to (a labelled delta under data_root with seed_paths declared and no --data-root typed lands under the first seed path), and on a run with no registered schema to import into.

Beside other findings the fork joins the same single envelope, with its own recourse.

Two cases raise no anchor finding. A project with no oplog skips the check, since import-driven flows write data files before any op exists. A record that belongs to a sibling schema under a shared data_root is left to that schema’s own data check.

A corpus holding a transition or cleanse record the project has not opted into refuses the run as experimental-not-enabled before any record is read, whichever files the operands name (Experimental purposes).

Refusal classes

A refused file’s line leads with its class, a name for what is wrong with the file (data/c.data.sql: writes-other-table record seq=1 writes relation "b" …), and the text refusal repeats the leading one as class:. In JSON the class is the discriminator of each entry under files, and a data-file-malformed envelope’s own discriminator names the most specific one, so a script branches on the name rather than the prose. A file with several faults is one entry, its detail naming each class.

Every corpus reader (data check, snapshot, migrate, data list) refuses these as data-file-malformed, and a producer refuses to write a file that would have one, except where a row names the one side that checks it:

classwhat it catches
grammarThe header line or the record fences break the data-file grammar: a missing, duplicated, unrecognized or out-of-order header token, a value outside its set, more than one plain record fence, or a delta section where none may stand.
unknown-formbody= or format= names no form this version admits.
manifest-reservedbody=manifest, which is reserved and not yet admitted.
fence-on-body-noneA body=none file holding a record fence.
format-on-body-noneA body=none file holding format=.
append-body-nonedisposition=append with body=none: an append adds rows, so it cannot declare a table empty.
unfilledA record still holding data scaffold’s <column> placeholders. data scaffold writes them; data check refuses the file, migrate and snapshot refuse it as dml-body-unfilled, and data list lists it. An append whose fence holds no records only warns data-file-unfilled.
unfilled-floorA floor (replace or restate) with body=data whose own section holds no records. data scaffold writes this shape; it is refused when read.
writes-other-tableA record writing a table other than the header’s table=, or more than one table.
replace-kindA replace record that is not an INSERT or a COPY supplying rows.
extension-mismatchThe file name’s extension disagrees with format=.
table-not-livetable= names no table live at the anchor. Checked when a producer writes the file and when data repin moves it.
purpose-outside-bandA floor whose purpose= is outside the band the fold admits.
ref-undeclaredA purpose=ref record holding rows, anchored where its table is not declared reference data (above).
ref-restate-declaredA purpose=ref restate body=none anchored where its table is declared (above).

A framing fault is one of the framing classes, also as data-file-malformed.

A file the reader cannot open is listed under files with class unreadable; when every refused file is unreadable, the run refuses data-file-unreadable instead, an envelope with no discriminator of its own.

A producer’s own refusals name a class too:

codeclasswhat it catches
data-file-malformedcascadeAn imported TRUNCATE … CASCADE: import cannot see which tables the cascade reaches, so it mints no floor for them.
data-file-malformedinterleaved-statementAn imported statement on a table between the statement that empties it and the INSERTs that refill it.
data-file-malformedoff-grainA file a producer would remove as superseded that holds more than one table, label set or anchor, or no longer parses; it is refused rather than partly edited.
data-file-collisionkey-collisionA write meeting a file it may not replace at its key: another key’s or another purpose’s record, hand-authored records, a file it cannot read, two pages landing on one path, or a floor no delta joins.
purpose-disposition-conflictpurpose-flag-outside-bandA --purpose outside the fold’s band on a run that would write a floor.

A fault between files keeps its own code when no file is malformed, and its envelope then has no discriminator: page-set-mismatch and anchor-unresolved still list each file under files with its class, while seq-claimed alone names the files only in its detail, so a script branches on the code. Alongside a malformed file, each joins the data-file-malformed envelope, every file’s entry under its own class:

codeclasswhat it catches
data-file-order-ambiguousseq-claimedTwo files at one anchor claim the same seq=.
data-file-page-setpage-set-mismatchA paginated record whose pages are not exactly 1 to N: a page missing, repeated or out of range, or pages disagreeing on N.
data-file-anchor-unresolvedanchor-unresolvedA record whose anchor-op= names no live op (above).

Framing classes

A record is one marker pair, and its body may hold one statement, several statements or one multi-row INSERT. Both marker lines are required, and every way to get the framing wrong refuses by name rather than mis-parsing:

classwhat it catchesremedy
end-missingA record’s declared end line never arrives before EOF.Close the record with its declared end line.
marker-in-bodyA marker-prefixed line inside a record body that was never closed: the next record’s marker read as body content.Close the first record with -- retrofit:dml-end, or escalate the frame: add end=<tag> to its marker line and close the body with -- retrofit:dml-end:<tag>.
end-line-whitespaceA line that differs from the declared end line only by trailing whitespace or a carriage return: an end line the scan cannot match.Normalize the file’s line endings; the body is otherwise correct.
unframed-contentContent between records that no marker pair frames, typically statements pasted straight into the section.Wrap the statements in a marker pair.
retired-bytes-tokenA marker line holding a bytes= token.Drop the token, and close the body with its end line.
bad-marker-tokenA marker line holding an unrecognized, duplicated or mis-ordered token, or a malformed end= tag.Correct the token to the record marker grammar.
duplicate-end-tagThe same end= tag claimed by two records in one file.Give the record a tag free across the whole file; the refusal names the lowest free tag.
byte-order-markA UTF-8 byte-order mark at the start of the file.Strip the BOM.
end-past-sectionA tagged record’s end line never arrives before the section end; the scan crossed the fence looking for it.Close the record inside its section.
empty-bodyA record whose body is empty: its marker line is immediately followed by its end line.Give the record a body, or remove the record.
section-end-in-minted-bodyWrite-path only. A body holding a -- retrofit:dml-section-end line that would be minted inside a section, where a tagged scan would swallow the fence’s terminator.Place the record outside a section.
section-end-in-bodyThe read-path mirror of section-end-in-minted-body: a record whose body consumed the enclosing section’s end line.Place the record outside a section.
duplicate-seqThe same seq= claimed twice within one file.Renumber so each record in the file claims a distinct seq=.

Every class but section-end-in-minted-body is a data check refusal as data-file-malformed, with the name as the refusal’s class and the offending line, so the refusal names the fix. section-end-in-minted-body is checked before any byte is minted, over the whole set at once, so a refused mint leaves no partial file behind; section-end-in-body covers a file that acquired that shape some other way.

A reference append (or restate) whose body is an UPDATE, DELETE, MERGE, INSERT … SELECT, or an INSERT leaving a column DEFAULT is legal and applies once in its window, but the chain no longer folds offline: drift cannot compare it and removed keys cannot be computed until the next version. data witness still witnesses the record and warns reference-chain-unfoldable naming the table, the record’s anchor and the statement that does not fold (structured: table, opid, reason); the exit is unchanged. A literal-INSERT append folds and raises nothing.

retrofit data witness

data witness [path...] binds reference records’ bytes to the oplog: it appends one WitnessReferenceData entry per selected record, over every file at its key. validate, op seal, migrate and snapshot verify a witnessed record whose anchor a seal covers.

retrofit data witness                 # every reference record in the unsealed tail without a witness at its bytes
retrofit data witness --dry-run       # the same rows, appending nothing
retrofit data witness corpus/public.colors.202609251400.data.sql

A directory operand refuses with witness-operand-directory. A named record outside the selection refuses with witness-band (a file that is not a reference record under data_root, or a record anchored outside its table’s reference span) or sealed-anchor (a seal covers its anchor). One already witnessed at its current bytes is reported already-witnessed. A data_root holding a transition or cleanse record the project has not opted into refuses the run as experimental-not-enabled (Experimental purposes).

retrofit data repin

data repin <file> --anchor-op <target> re-points one corpus data file’s anchor-op= at an opid you name: the reviewed recourse of a record whose anchor names no live op (data-file-anchor-unresolved, above), or whose row shape moved at an anchor that still resolves (anchor-shape-mismatch).

retrofit data repin corpus/public.colors.202609251400.data.sql --anchor-op last --dry-run
retrofit data repin corpus/public.colors.202609251400.data.sql --anchor-op 4c1382aed592

The target takes migrate --to’s grammar, and what is written is always the resolved 12-hex opid. Without --anchor-op the verb refuses anchor-target-required. Run retrofit op show to read the opids.

A file is eligible when its anchor names no live op, or when its anchor resolves but the row shape there no longer matches its anchor-shape= token (or the token names a version or algorithm this binary cannot read). A file whose anchor resolves with a matching shape, or with no token, refuses anchor-still-resolves; capture the rows at the new point with data capture instead. A moved shape at an anchor a seal covers refuses sealed-anchor, its hint naming op unseal first. A file already at the target is already-pinned, exits 0 and writes nothing when its shape matches; when the shape moved, it is re-stamped at that anchor and reported restamped.

A re-pin rewrites the header’s anchor-op=, writes the anchor-shape= token at the target when the body has a shape, and re-stamps each record’s seq= after the records already at the target, as a mint there would. Every other byte of the file (body rows, comments, record markers and the other header tokens) is passed through unaltered. --dry-run computes and reports the same decision, prefixed would repin: , and writes nothing.

A re-pin of a witnessed reference record re-witnesses it: the verb appends one WitnessReferenceData entry at the new anchor, over the rewritten bytes, and names its opid. The old witness is left in place. An unwitnessed record stays unwitnessed, and --dry-run appends nothing.

The target is also checked for legality before any byte is written, and each of these refuses with the file unchanged:

TargetRefusal
The cut before the first op (root).anchor-target-pre-genesis
A cut where the file’s table is not live.data-file-malformed, class table-not-live
A cut where a ref record’s table is not declared reference data. A body=none floor is exempt, since it holds no rows.anchor-target-undeclared
A cut where a ref restate body=none floor’s table is declared.data-file-malformed, class ref-restate-declared, the refusal data check would give the moved file
A cut another file already claims for the same table and labels.data-file-collision
A cut a seal covers, for a record that is not purpose=seed.sealed-anchor

A paginated file refuses with data-file-paginated before any of these unless --anchor-op is the page’s own full 12-hex anchor-op=, which re-stamps that page where it stands. The verb does not move a page set, so re-capture it at HEAD instead. Between that check and the target’s, a transition or cleanse file, or a data_root holding one, refuses with experimental-not-enabled unless the project opted into its purpose (Experimental purposes).

A seed record re-pins into a sealed range, and the run names the seal that covers the target (seed-anchor-sealed).

retrofit data reanchor

data reanchor [<file> ...] re-anchors records under data_root onto the anchor that now holds their place, and named files narrow the sweep to those records. op revise, op remove, op unseal and a single-file reconcile run the same sweep after their own write and print its report.

retrofit data reanchor --dry-run
retrofit data reanchor

Two kinds of record move:

  • A retired anchor. A record of any band whose anchor-op= the oplog revised or removed moves to the replacing op, or to the op before a removed line, followed until it names a live op. An oplog in the compact form keeps no trace of a retired op, so there the record is listed for data repin; one on a removed first line, which has no op before it, is listed for a capture at HEAD.
  • A release behind it. A reference record whose anchor a seal covers, and which has no witness in that seal’s range, moves onto the newest Seal line. A seed record is never moved this way, and a witness whose digest no longer matches is an integrity finding, never moved.

A record moves only when its anchor-shape= equals the row shape at the target; a mismatch, or no token, is listed. A move rewrites anchor-op= and re-stamps seq= as a mint at the target would. Each witness of the record moves with it, its digest kept, and is stale until retrofit data witness; the report then ends with a note naming data witness before op seal. Run alone or by reconcile, a record whose target key another record already holds is listed as a fork, never merged.

The report is one line per record, in file-path order:

reanchored: <file>: <table>@<from> → @<to> (<cause>): <clause>
listed: <file>: <table>@<from>: <reason>; <recourse>
fork: <file>: <table>@<from>: <reason>; <recourse>
not selected: <file>: <table>@<from>: <reason>

<cause> is revised, removed or release <name>. not selected: is a named file the sweep does not pick. Listed records exit 0. --dry-run prints would reanchor: for reanchored:, every other line unchanged, and writes nothing; a sweep with no line prints nothing to reanchor. Under --as=json each line is one record with file, from, to, cause, outcome (reanchored, listed, fork or not-selected), reason and recourse on a record that did not move, holder on a fork, and dry_run: true on a rehearsal.

A named path that is not a readable record file refuses data-file-unreadable, or data-file-malformed for a readable file that is not a record, before any write. When the sweep fails after a calling verb’s write, the write stands and the run exits 1 naming retrofit data reanchor; a rerun completes it.

retrofit schema ref-tables

ref-tables declares which tables hold reference data (rows the application reads but never writes, such as currency codes, country lists and status enumerations), applied on every emit with no --label. It is the first step of the classification chain in Purpose, and the declared set is the reference capture scope every producer run reads at its own minting cut: import, data capture, drift (which captures the declared tables at its new head on every run, with no flag), and data scaffold.

add and remove append one SetReferenceData op per resolved table to the oplog. A table can also be declared as it is created (op add CreateTable … reference_data=true, or --ref <table> on import and drift), and ref-tables reads those declarations like its own.

retrofit schema ref-tables                          # read the declared set and report
retrofit schema ref-tables add countries 'curr*'    # declare the tables the arguments resolve to, then report
retrofit schema ref-tables remove currencies        # withdraw a declaration, then report

All three forms print the same report, the whole declared set rather than a delta, so a read and an edit compare byte for byte.

Each argument is an exact table name or a glob, resolved against the tables live at the read cut: HEAD, unless --at <opid> names another. A glob expands when you run the command, so a table created later is not declared by an earlier glob, and a glob never declares an attached partition child: it reports each one as skipped: instead. An argument that resolves to nothing is refused (ref-table-not-live): import the DDL, drift it in, or create it with reference_data=true, then declare it. Declaring a table that is already declared, or removing one that is not, is a per-name no-op that mints nothing and says so (already-declared, not-declared) rather than refusing; --ref and op add SetReferenceData answer the same way. The exception is a merge or join product whose sources disagreed about the declaration: op seal refuses it until a SetReferenceData on it records the judgment, so every one of those writes mints that line even though the product already reads declared.

The report lists the declared set as one declared: line per table, in bytewise order. On add and remove one line per resolved table follows: the table name, then the SetReferenceData op the run minted (or, under --dry-run, would mint) as its on-disk oplog entry verbatim, SetReferenceData(<opid>, prev=<prev>, on=<element-id>, value=<bool>), the same line that lands in the oplog. It is a record of what was written, not a command to run. Then comes one line per disagreeing table, and a run-level summary:

$ retrofit schema ref-tables add countries 'curr*' --schema app
declared: countries
declared: currencies
countries  SetReferenceData(00664d8eaeaa, prev=6ccda870b705, on=081f8bafa7ea, value=true)
currencies  SetReferenceData(32bbc6fd0f0e, prev=00664d8eaeaa, on=2513d190b2ad, value=true)
ok  app  declared_tables=2  disagreeing_tables=0  dry_run=false
$ retrofit schema ref-tables add countries loan_periods --dry-run --schema app
declared: countries
declared: currencies
declared: loan_periods
countries  already-declared
loan_periods  SetReferenceData(8c78aa7cd7e2, prev=32bbc6fd0f0e, on=49448cf0ecb0, value=true)
ok  app  declared_tables=3  disagreeing_tables=0  dry_run=true

A run with --at appends at=<opid> to the summary. The full grammar, text and JSON, is the schema ref-tables row of Output contract. --data-root moves the base corpus the disagreement report reads; it adds no seed layer.

The first declaration in a project whose data_root is the undeclared default and does not exist yet creates that root with an empty .gitkeep, and says so on stderr, so every checkout has the root the declaration’s records will land in:

created data/.gitkeep so every checkout has the data root; keep it with the reference-table declaration

Commit the file with the declaration. A root you named (the data_root key or --data-root) is never created.

Refusals

Argument failures accumulate: every argument is evaluated and each failure is reported beside the argument that caused it, rather than the run stopping at the first. The envelope has the covering code ref-tables-rejected, each entry in its violations array has its own per-failure code (see Output contract), and the whole invocation is refused, so nothing is minted, not even for the arguments that resolved.

CodeWhen
ref-table-not-liveThe argument resolves to no table live at the read cut: an exact name no live table has, a glob that matches none, or a glob whose every match is an attached partition child.
ref-table-pattern-invalidThe argument is not a well-formed glob (an unterminated character class, say).
ref-table-pattern-qualifiedThe argument qualifies itself with the selected schema’s own name. The schema is already selected (by --schema, or from the config chain when it is omitted), so the argument takes a bare name.
reference-data-partitionThe argument names an attached partition child exactly. The refusal names its partition root; declare the root instead.
ref-tables-rejectedThe covering envelope code holding the array above. It names the envelope’s kind, never the first failing argument’s.
ref-tables-schema-not-in-cutThe named schema is registered in retrofit.toml but absent from the oplog at the read cut, so there is no live table universe to resolve against. Distinct from unregistered-schema, which is the registry miss.

An unregistered schema refuses with unregistered-schema. Each code’s exit status is on Refusal codes.

Notes

Three advisories follow the report at the courtesy band (they drop under -q): dml-ref-table-no-match-universe (no table is live at HEAD, so an argument is neither declared nor refused), dml-ref-table-policy-disagreement and dml-ref-table-seed-dormant (Advisory codes). An add (and add --dry-run) that moves a table holding a seed chain into a reference span, in the base corpus or a seed_paths root, raises one dormant note per chain, naming its label set and earliest anchor; its hint names the chain’s recourse, as listed under data list above. op add SetReferenceData <table> true and import/drift --ref on a live table raise the same note.

The data file

Custom DML (INSERT, UPDATE, DELETE and MERGE) is outside retrofit’s vocabulary, so no op records it. import writes each such statement, verbatim, to a runnable file per table instead: the data file. data capture, drift and data scaffold write the same files. The corpus is kept outside the oplog (Data files explains how it follows the schema). snapshot and migrate read every addressed *.data.<format> file under data_root on every run (The data band, Preserved DML).

A run that writes custom DML writes one data file per table:

  • Name: <schema>.<table>.<YYYYMMDDHHMM>[.p<K>].data.sql, the table and then the UTC minute the run captured it at. The stamp is opaque and never read back.
  • Location: the schema’s data_root, a [[schema]] field in retrofit.toml that schema create scaffolds; unset, it is [project] data_root, then data under the project root. A labeled record is in a directory named after its first label in sorted order (data/<label>/…). A label, schema or table name that is not only lowercase letters, digits and _ is written folded to those characters with a short digest after a -, so the label seed:x is the directory seedx-<digest>.
  • Seed roots: seed_paths (in [project], or in a [[schema]], where it replaces the project’s list; a [[schema]]’s extra_seed_paths adds to it) names further roots holding seed records, and a seed record is written under the first entry when one is declared. --data-root, --seed-path and --extra-seed-path replace data_root, replace seed_paths and add a seed root for one run.
  • Key: {table, band, label-set, anchor}, stated in the header. A record of another band (ref, seed, or the history band that transition and cleanse share) at the same anchor is another key and is left standing, so an import --purpose transition beside a reference record is a second file, never a rewrite.
  • Re-capture at a held key: a floor is overwritten, an append accumulates (at a floor’s key, as the floor’s delta section), the floor’s own source moves it, a witnessed key refuses data-witnessed-key, and identical records write nothing. See Re-ingesting into the corpus.
  • Name collision: a computed name already held by a record of a different key takes the next minute, and the next, until the name is free, so two records of one table written in one minute are two files.
  • Readers use the header: every fact comes from the header, never the name, so a moved or renamed file still resolves.

One table per file is the grain to curate by: rm data/<label>/c.* removes table c at every anchor. Two shapes hold more than one file for a table: a capture whose record count exceeds the page bound is a page set (.p2, .p3, … with page=K/N on each header), and a floor that a later append at its anchor does not join (a hand-authored floor, or a scaffold’s empty floor) stands beside that append’s own file. Removing one member of either does not curate the table out (the sibling still speaks for it), so curate the set, and let retrofit data check confirm the corpus reads cleanly before you commit.

No producer runs a data file. The rows reach a database only when you apply them: run the file yourself, or apply the SQL snapshot or migrate emits, which holds the bodies.

File layout

Each file is line-oriented. A v1 header tuple names the source, the one table the file declares, the anchor it is pinned at, and the file’s axes (purpose, disposition, body, format, labels, and page on a paginated capture). Every record is inside the file’s single body fence: a marker line, its verbatim text, and an end line that closes the body. The body is written exactly as captured, with nothing appended: a statement that arrives without a trailing ; is stored without one, and migrate supplies the terminator when it renders:

-- retrofit:data-file v1 source="01_seed.sql" table=["public","users"] anchor-op=988edcb612ab purpose=seed disposition=append body=data format=sql labels="[\"env:dev\"]"
-- retrofit:dml-section
-- retrofit:dml seq=1 src-file="01_seed.sql" src-line=42 kind=InsertStmt
INSERT INTO users (id, email) VALUES (1, 'a@example.com');
-- retrofit:dml-end
-- retrofit:dml seq=2 src-file="01_seed.sql" src-line=44 kind=UpdateStmt
UPDATE users SET active = true WHERE id = 1;
-- retrofit:dml-end
-- retrofit:dml-section-end

A floor’s file may hold a second, delta section after the floor’s own fence: a delta that meets the floor at its key (an import after a data capture, or a script’s UPDATE after its TRUNCATE; INSERT) joins the floor’s record rather than starting a second file. Its fence opens -- retrofit:dml-section delta, the one token a fence admits; the header keeps disposition=replace, its records are deltas (any DML, each naming its own src-file), and migrate loads the floor, then runs them:

-- retrofit:data-file v1 source="fixtures/track.sql" table=["app","track"] anchor-op=0123456789ab purpose=seed disposition=replace body=data format=sql labels="[\"qa\"]"
-- retrofit:dml-section
-- retrofit:dml seq=1 src-file="fixtures/track.sql" src-line=1 kind=InsertStmt
INSERT INTO app.track (id, user_id, status) VALUES (1, 42, 'open');
-- retrofit:dml-end
-- retrofit:dml-section-end
-- retrofit:dml-section delta
-- retrofit:dml seq=2 src-file="refund-case.sql" src-line=1 kind=UpdateStmt
UPDATE app.track SET status = 'refund_failed' WHERE id = 1;
-- retrofit:dml-end
-- retrofit:dml-section-end

A body=none file is a floor (it declares the table empty at its anchor) and has no fence at all: the header is the whole file. A delta that meets an empty floor at its key (TRUNCATE t;, then an INSERT, in one script or a later import) joins it the same way: the record is rewritten body=data with an empty floor fence, then the delta section, and migrate clears the table, then runs the delta.

The body is the whole lines between the marker line and the end line. The newline just before the end line is part of the framing, not of the body, so a body never gains a trailing newline its author did not write. Everything in between is kept byte for byte: no escaping, no unescaping, no normalization.

Header fields

The retrofit:data-file and retrofit:dml marker tokens are the on-disk grammar and stay stable for re-ingest. The header tuple states the file’s facts in this fixed order:

  • source: where the file came from (the import source, a drift dump, or scaffold). Provenance only; nothing checks it.
  • table: the one table the file speaks for, as a two-element JSON array ["<schema>","<relname>"], each half spelled exactly as the oplog spells it. Every record body must write that table and no other (class writes-other-table); a record that writes another relation belongs in its own file.
  • anchor-op: the 12-hex opid of the op appended immediately before the statements’ source position. migrate splices the bodies back in just after that op’s DDL. A statement with no DDL before it in its source file falls back to that run’s first op; a data-only import (no DDL) anchors at the oplog’s last op instead. An authored statement whose table does not stand at its anchor refuses with dml-table-not-found, and an empty oplog models no table, so a data-only import that captures a statement into one refuses that way (a statement no step of the purpose chain classifies is listed, not captured). Statements anchored at different ops are in different files.
  • anchor-shape: a digest (v1:sha256:<hex>) of the columns the record writes, as they stand at the anchor. data check lists a record whose digest no longer matches (anchor-shape-mismatch), and snapshot and migrate refuse it; data repin re-stamps it.
  • purpose: the classification (see Purpose). Absent means unclassified.
  • disposition: append, replace or restate. replace declares the file is the table’s full observed row set at its anchor; import infers it from a TRUNCATE or unqualified DELETE: the INSERTs for that table that follow the arming statement are written as one replace file, and an arming statement with no following INSERT writes a replace body=none floor. Everything else is append; import --disposition sets it for the whole run.
    • A statement the source states before the arming statement, at the floor’s own anchor, is wiped by it: the floor supersedes it, nothing is written for it, and the run names it dml-superseded: <src-file>:<line> (<schema.table>; superseded by the arming statement at <src-file>:<line>) ({"type":"dml-superseded","table":…,"src_file":…,"src_line":N,"arming_file":…,"arming_line":N}). A statement at another anchor (DDL between) keeps its own record, and one that also writes another table (a data-modifying CTE) is never superseded: the write refuses its second relation instead.
    • A non-INSERT on the table stated after the arming statement but before or among the INSERTs that follow it, at the floor’s anchor, is neither a floor row nor a delta that follows the floor, so the import refuses before writing anything (dry run included), naming the arming statement, that statement and the first INSERT after it. Replace the arming statement with a qualified DELETE FROM t WHERE true; (which never arms the window) so the script is a delta, or move the INSERTs ahead of the statement; one stated after the last INSERT is the floor’s delta section.
    • restate (hand-authored or scaffolded only) restates the table’s rows for exactly the file’s label set, ignoring that chain’s earlier records.
    • A replace or restate file must have a fold-band purpose (seed or ref). Records of one (table, label-set) chain are versioned by their anchor-op position: the latest floor at or before a render point supersedes earlier state, while snapshot --at still renders older versions.
  • body: data (the fence holds the rows) or none (a floor declaring the table empty; no fence). Absent means data.
  • format: sql, the inline body’s serialization; written on a body=data file only, and the file’s extension must agree with it (.data.sql).
  • labels: the optional label set (see Labels), written only when the file has at least one label.
  • page: K/N on a paginated capture only.

Record markers

Each record marker inside the fence has:

  • seq: a 1-based counter in capture order, one stream across every file a run writes, so records merge back into source order across files. Two records at one anchor with the same seq in different files are an order ambiguity, refused with data-file-order-ambiguous.
  • src-file (relative to the project root) and src-line: where the statement came from, JSON-quoted like source. A hand-authored record has src=hand instead.
  • kind: the statement’s parse-tree node kind (InsertStmt, …). A replace file admits InsertStmt (and hand-authored CopyStmt) records only.
  • end: the optional framing tag (see Escalated framing). Present only when the body needs one; it names the end line that closes this record, and it is the marker’s last token.

Escalated framing

The default end line is the fixed text -- retrofit:dml-end. That works until a body itself contains a line starting with -- retrofit:dml, at which point a reader could not tell the body’s line from the real frame. So:

A record must use escalated framing whenever any line of its body begins, at column 0, with the literal -- retrofit:dml. Escalated framing adds an end=<tag> token to the marker line and closes the body with -- retrofit:dml-end:<tag> instead:

-- retrofit:dml seq=3 src-file="03_notes.sql" src-line=7 kind=InsertStmt end=1
INSERT INTO notes VALUES ('a literal
-- retrofit:dml-end
inside a string, so this record escalated its tag');
-- retrofit:dml-end:1

A tagged record is closed only by its own tagged end line. Inside one, every other marker-like line (including -- retrofit:dml-section-end) is ordinary body text, so escalated framing is how a body holds lines that look like markers.

Choosing a tag:

  • A tag is a decimal integer with no leading zeros (1, 2, 17).
  • Tags are unique across the whole file, not per record and not per section. The tag assigned is the lowest one still free in the file, in record order, so the first escalating record usually takes 1, the second 2, and so on. A record that does not escalate uses no tag, which is why a file may hold end=1, then an untagged record, then end=2.
  • A tag must also be absent from the body it closes: if the body already contains the line -- retrofit:dml-end:<tag> at column 0, that tag would close the record early, so it is skipped. Tag allocation may therefore skip integers: a body containing -- retrofit:dml-end:1 gets end=2 even when 1 is otherwise free.
  • When you add or copy a record by hand, pick a tag that is both free across the entire file and absent from that record’s body. Reusing one fails the next read with data-file-malformed, and the message names the lowest tag still available across the file; if that tag collides with a line in your body, take the next one that does not.

The literal prefix is -- retrofit:dml and nothing wider: the -- retrofit:raw, -- retrofit:snapshot-data and -- retrofit:data markers belong to other features and are ordinary body content, so they never force escalation.

One placement rule follows: a record whose body contains a column-0 -- retrofit:dml-section-end line cannot be placed inside a section, because its body would swallow the section’s own terminator. That shape fails with data-file-malformed on write and on read. The same body outside a section is fine.

If a data file will not parse

A framing fault fails the read with data-file-malformed, naming the record and line under a class that names the fault, such as end-line-whitespace or marker-in-body (every class). The common ones:

  • A body’s end line is missing, or was damaged; a CRLF rewrite is the usual cause, since end-line recognition is byte-exact. When only trailing whitespace differs, the message says so, because the fix is your editor’s line-ending setting rather than the body.
  • A body contains a -- retrofit:dml line but the record was not escalated. The remedy is the tag escalation above.
  • Anything other than blank lines and ordinary -- comments sits between one record’s end line and the next marker line.
  • A record’s body is empty.
  • A marker line has a bytes= token. Delete it from each marker line and close each body with a -- retrofit:dml-end line.

Purpose

Each record has a purpose, which tells the emit commands what to do with it; What a data file is for has a table of the five. Beside that table:

  • transition and cleanse are experimental: a project uses them only by opting in (Experimental purposes). cleanse is emitted under a run-once marker.
  • ref is never label-filtered.
  • A seed record no --label names is excluded and counted, cause not-selected.
  • unclassified is the fail-closed default. A header without a purpose= token reads as it, and data scaffold writes it when no purpose, label or policy match applies; import sets aside a statement it cannot classify rather than write it. migrate refuses an unclassified body unless you pass --allow-unclassified-dml; snapshot excludes and counts it.

import classifies each statement it preserves once; the first step that applies decides. data capture and data scaffold classify by their own rules.

  1. The reference-data declaration, an attribute in the oplog, not configuration. A literal INSERT the reference fold admits (an ON CONFLICT clause, for one, fails it) into a table whose read at the run’s cut is reference_data=true is ref, whatever --purpose or --label says.
  2. --purpose transition or --purpose cleanse: a run-level assertion stamped onto every other statement the run preserves. --purpose takes no other value, and is not combined with --label. Either value needs the project’s opt-in (Experimental purposes); without it the run refuses before it writes anything.
  3. A declared table, otherwise: any other statement into a declared reference table is listed with cause no rung; it is never seed and never ref.
  4. --label: a run that names one or more labels writes its records as seed, with the labels stamped on them. A label names a dataset, and a dataset is a seed band, so the band follows from the label.
  5. Otherwise, the statement is listed and written to no data file: an authored statement with cause no purpose or label, a dump’s data band with cause not declared. There is no default transition or seed step: a purpose must be asserted, target a declared reference table, or follow from --label.

Experimental purposes

transition and cleanse are experimental (Data files says why). A project uses either purpose only by opting in, in retrofit.toml:

[experimental]
purposes = ["transition", "cleanse"]

The table is project-wide, and the opt-in is per purpose: a project that lists only transition still refuses a cleanse record. A name other than the two, a name listed twice, a purposes that is not an array of strings, or any other key in the table refuses at load as config-invalid.

Without the opt-in, both purposes refuse as experimental-not-enabled, before anything is written and under --dry-run too:

  • import --purpose and data scaffold --purpose refuse to write a record of either purpose.
  • validate, import, data scaffold, data list, data check, data witness, data repin, schema ref-tables, migrate and snapshot refuse a data corpus holding one, before any record is read. The check reads every data file’s header, whatever the run applies, so data check <file> and snapshot --ddl-only refuse too. The detail names each file and its purpose, one <file>: purpose=<purpose> line per file.

The refusal’s hint names the key and the alternative below. With the opt-in, each verb reads and writes these records as it does any other, and prints an experimental-purpose warning once per such file it reads or writes. The warning does not change the exit.

The alternative is a backfill written as a DO block. import and op add record it as a RunRawScript line in the oplog, which a Seal’s digest covers. Declare its reversal on the same line; here reverse_noop says a down needs no action:

retrofit op add RunRawScript \
  'DO $$ BEGIN UPDATE public.orders SET total = 0 WHERE total IS NULL; END $$;' \
  'reverse_noop=the backfilled totals stay after a down'

With no reversal declared, a down over the script refuses, and import and op add report reverse-undeclared when they record it (The reverse migration).

Labels

--label <label> (repeatable) names a dataset for the run: step 4 of the purpose chain. Labels select cohorts within the seed band, so only seed records have them: a record step 1 or 2 classified ref, transition or cleanse, or that the chain lists, is never labeled. The value is free-form (a key:value form is accepted, not required); labels are sorted and deduplicated on disk. They mean nothing to retrofit; they let you name the dataset at emit time with --label <label> on snapshot or migrate.

Re-ingesting into the corpus

A re-ingest whose records anchor at a new op writes a new data file. At a key that already holds a record:

  • Identical records write nothing.
  • A divergent floor (disposition=replace or restate) at an unwitnessed key replaces that key’s version as a set: matched files are rewritten in place under their original names, surplus pages are deleted and named in the report, missing pages are written, and each replaced file is reported overwrote: with its digest. Only a commit keeps the prior bytes.
  • A floor at an append’s key (--disposition replace where no floor stands) overwrites the standing record, every source’s statements, when its rows differ; another source’s equal rows leave the record as it stands. Each replaced file is reported overwrote: with its digest. A src=hand record there refuses data-file-collision.
  • An append accumulates: the incoming statements join the standing record after its own, reported data-appended: with the digest and the statements added. At an unwitnessed floor’s key they join the floor as its delta section, reported data-appended: … (replace; delta <src> added, +N statements); an identical floor re-capture leaves the record as it stands, and a divergent one replaces it whole. Two data-only imports into one table add no op, so they share the anchor and leave both sources’ statements in the one record, in import order.
  • Accumulation is per source file. Re-importing an unchanged src-file changes nothing, and re-importing it changed replaces that source’s statements where they stand rather than adding them again, so it still replays before the sources imported after it. A changed source that now holds more statements than before keeps its place too: the extra statements take the next seqs after its section, and the record’s later statements shift up by the growth, in order, reported N shifted (<src-file>). Seqs an earlier shrink freed are room. When the shift would reach a seq another file holds at the anchor (another label set’s record, say), the import refuses data-file-collision naming that file and seq before anything is written: another key’s file is never re-stamped, and the section is never moved to the end.
  • Statements across sources. A changed source’s old statements go even when every new one is a duplicate another source holds. A statement restating a keyed row another source states supersedes that source’s statements. The header’s source= stays the first writer’s, each statement keeps its own src-file and src-line, and a statement another source already holds is skipped as a duplicate.
  • An append of a different purpose than the standing record’s is not merged into it and overwrites it.
  • A witnessed key refuses data-witnessed-key.

Importing the source a floor was captured from (data capture s.sql, then import s.sql) moves the floor rather than deduplicating against it. The last command run on a source decides what the key says about it, whatever the bytes, so the floor’s file is removed and s.sql’s statements become the key’s append (replace → append, joining a standing append there, if any, and keeping the floor record’s other delta sections). A run that also writes a floor at the key moves nothing. The report names the clear step migrate no longer renders and the way back:

moved ref: zip - s.sql is now a delta at <anchor>, not the whole table (replace → append)
  note: rows the table held that s.sql does not are no longer deleted at migrate
  to restore the whole table: retrofit data capture --schema app -- s.sql

({"type":"data-moved",…} with table, anchor, labels, src_file, from, to and path under --as json). Only the floor’s own source moves it: another source restating the floor’s rows is still the silent duplicate skip, and a source from stdin has no identity to move by. The way back holds at identical rows too: data capture s.sql on the moved key writes the floor again and reports the move back (overwrote @<anchor> (append → replace), or supersedes once HEAD has moved; see retrofit data capture). --dry-run reports the move as would move … and writes nothing.

--purge-superseded opts out of keeping superseded cross-position blocks, except a witnessed record, sealed or in the tail, which the purge never touches: each is left byte-intact and reported as kept: <schema.table>@<anchor> <labels> <path> (witnessed by <opid>; …) ({"type":"data-file-kept",…} under --as json).

Before 1.0, any release may change behavior; the release notes list what changed.

Published .