retrofit data: the data corpus
The data verb group works on the data corpus: every addressed
*.data.<format> data file under a schema’s declared
data_root (written as .data.sql). Data files hold row data
(reference tables, fixtures, migration backfills) beside the DDL history
the oplog records, and the emit verbs (snapshot,
migrate) fold them into a data band at emit time.
Each verb’s flags are on its own page in the Command reference, which
the table links.
| Verb | What it does |
|---|---|
data list | Show the fold the emit verbs would apply. Read-only. |
data capture | Capture observed rows from an INSERT-style dump. |
data retire | Withdraw one table’s data chain at the current head. |
data scaffold | Mint a data file for a table holding one inert placeholder record to fill. |
data check | Validate the corpus against the data-file grammar. |
data witness | Bind reference records’ bytes to the oplog. |
data repin | Re-point one data file’s anchor-op= at an opid you name. |
data carry | Move records whose anchor was revised, removed or sealed past onto the op that now holds their place. |
schema ref-tables | Read or edit which tables hold reference data. |
Synopsis
From retrofit data --help:
Usage:
retrofit data
retrofit data [command]
Options
From retrofit data --help:
Flags:
-h, --help help for data
--help-all show extended help (vocabulary, semantics, caveats)
The global flags are in the Output contract.
retrofit data list
data list is the consumer-side dry run of the data band. It answers
“what would snapshot / migrate actually apply?” over the same fold
those verbs read, and it emits no SQL and writes nothing.
retrofit data list # the reference fold, then the label inventory
retrofit data list --label env:dev # the fold for one dataset
retrofit data list --label base --label env:dev # two layers, in that order
retrofit data list --label env:dev --at op48 # the fold at an earlier cut
retrofit data list --schema sales --label env:dev # a non-default schema
Work with row data walks a run with no --label and a run
that names two datasets.
The fields on a line
data list --help-all defines each token of an item line. Beside it:
- Items are one line per table, in bytewise table order. An item line is
the table, its
state(below), thenfloor=,witness=,removed=,appends=,excluded=ands3=in that order;witness=andremoved=follow a floor only. - A bare run (no
--label) folds the reference band only, counts each labeled chain’s records undernot-selected, and ends with onelabel <label> files=Nline per label, in bytewise label order. witness=readswitnessed <opid>when a witness governs the floor record and its bytes still match,witness-stalewhen they no longer do, andunwitnessedwhen no witness governs it. Witnesses govern reference records only, so every seed floor readsunwitnessed.--labelis ordered: each occurrence is one layer, and a later layer’sreplacefloor supersedes the layers below it for that table. A file holding several named labels folds in each of those layers and is emitted once, at the highest of them.- When the run repairs a record for a schema change since its anchor (a
rename, say), a
notice: <table>: …line on stderr precedes the items. - Under
--as=json,appends[],excluded[]ands3[]name each entry the text line counts. Ans3[]entry’slower_labelis the layer of the floor it is anchored below, asfloor=spells it (reffor the reference band).
s3=1 below says env:dev holds one contribution to public.accounts
anchored before base’s floor at op61; --as=json gives its anchor:
$ retrofit data list --label base --label env:dev
public.accounts occupied floor=base/replace/data@op61 witness=unwitnessed appends=1 excluded=0 s3=1
state: what the fold decided
Each item has one state literal:
state | Meaning |
|---|---|
occupied | A floor selected, and rows survive to the emitted band. |
empty | A floor selected, and it declares the table empty (body=none). |
composed | No single floor; the band is composed from appends alone. |
unspoken | The corpus holds records for this table, but this run’s labels named none of them. |
none | No record in the corpus addresses this table at all. |
dropped | The table was dropped from the schema, or ended by a JoinTable, SplitTable or MergeTable; its records can no longer apply. |
A dropped table’s records are listed in excluded[] under table-dropped.
A table a JoinTable, SplitTable or MergeTable ended takes that op’s
own cause instead (table-joined, table-split, table-merged,
table-inlined or table-unjoined). Nothing is owed on the source’s
records: they stay as history, and a witnessed one stays verifiable. A
declared product is captured on its own: op add advises the capture
(data-motion-product-capture-owed), and op seal refuses
reference-rows-uncaptured until the product has a reference record
anchored at or after the op.
A seed record on a table inside its reference span is dormant: while
the table is declared at the render point, data list, snapshot and
migrate never fold its seed records, stack them over the reference rows, or
apply them alone, whether or not the table holds reference records yet. A run
that names the chain’s label lists them in excluded[] under
reference-data, and a table whose only named records are dormant reads
unspoken, never none. The seed chain is kept and never reclassified.
The emit verbs warn once per dormant chain (apply-risk-data-dropped,
cause reference-data, at every verbosity), naming the chain’s recourse
by the table’s name at HEAD:
- For a one-label chain,
retrofit data capture --from-label <label> --table <table>promotes the fixture’s rows into the reference chain. - For a chain with several labels, which
--from-labelcannot fold, the warning names a capture of the table from a database instead:retrofit data capture <dump> --table <table>. retrofit data retire --label <label> <schema.table>withdraws a chain that was only a stand-in; for a chain with several labels, the line names every label.- When HEAD no longer has the chain’s table (a historical
--atinside a span whose table was later dropped), the warning names no command.
A --schema you typed is repeated on the capture line. Outside every span,
before the declaration or after schema ref-tables remove, the seed chain is
an ordinary dataset again.
Refusals
data list refuses where snapshot’s corpus scan and fold refuse, with
the same code, and never prints a partial listing. A declared
data_root that does not exist refuses data-path-missing, and a
corpus holding a transition or cleanse record the project has not
opted into refuses experimental-not-enabled
(Experimental purposes). An unreconciled
fork between the first op and the cut refuses
fork-in-range, as snapshot does; an --at at or before the
fork’s shared prev= lists normally. Refusal codes
gives each code’s exit status.
No witness finding refuses data list. Where validate, op seal,
snapshot, migrate and data check refuse a covered reference record,
data list lists the record and reports the finding in its witness=
token instead:
| Finding the other verbs refuse | data list reports |
|---|---|
data-witness-mismatch: the record’s bytes differ from its witness | witness-stale |
data-witness-missing: no witness in the covering seal’s range | unwitnessed |
witness-out-of-range, witness-digest-unsupported: no witness governs | unwitnessed |
data-file-missing: a witness names a record no file holds | a stderr line listing the record from the oplog, witnessed <opid> |
A record in the unsealed tail whose witness is stale or absent reads
witness-stale or unwitnessed too; there only op seal refuses it.
To get the refusal and its code, run retrofit validate or retrofit data check.
JSON
Under --as=json each item is a {"type":"item", …} record on stdout,
followed by the terminal {"type":"ok"}; a bare run emits one
{"type":"labels","present":[…]} record after the items and before
{"type":"ok"}. See Output contract for the record
shapes.
The -- Data: designation line
snapshot and migrate write one comment line above every run of rows they
emit, naming the rows’ band:
-- Data: "shop"."statuses" (reference)
-- Data: "shop"."fixtures" (seed, label qa:golden)
The designation is reference, seed, label <label>, transition, cleanse
or unclassified. A multi-label chain’s line names its placing layer’s label:
the highest-positioned --label that selects the chain. Dropping that label
from argv still ships the rows when another named label selects the chain,
and the line then names the other label. A labelled ref record reads
reference whatever its labels. A replace block that also holds another
label’s rows opens that label’s line above them, inside the block, and its own
line again below. data capture drops these lines from the bodies it mints
and keeps any other comment, including a hand-written
-- Data: loaded by hand.
retrofit data capture
data capture <source> records what an INSERT-style dump says each table’s
rows are at the current oplog head. It observes state, so it has no
--purpose flag.
retrofit data capture rows.sql --label env:dev
retrofit data capture rows.sql --table countries --table 'curr*'
retrofit data capture rows.sql --schema sales --dry-run
A table whose oplog declaration reads reference_data=true at the run’s
cut is captured as ref; otherwise a non-empty --label set captures it
as seed with exactly that label set; otherwise the table is listed, not
captured, and the run exits 0. There is no producer default.
--from-label <label> reads a label’s seed fold instead of a source
(Capturing from a corpus label).
Capture reads INSERT statements, including the output of
pg_dump --inserts and pg_dump --column-inserts. COPY ... FROM stdin,
UPDATE, DELETE and MERGE are not capture input; import takes
authored DML.
Every captured table is a full observed row set (disposition=replace) at
HEAD, and the table’s key at render sets how that replace replays (The
replace prologue). A table
with no key the rows write in full is cleared whole-table and reloaded
when no other table’s foreign key references it, and refuses
key-uncovered when one does. A secondary UNIQUE the arbiter does not
cover, or a surrogate key re-minted at capture, can make an upsert
collide or replace rather than update at apply. The recourse is chosen at
capture time: declare the constraint DEFERRABLE INITIALLY DEFERRED
(which removes it from arbiter candidacy; if the key is then a unique
index on the same columns, the render refuses replace-no-upsert-form),
or, for a table no other table’s foreign key references, capture the
table keyless.
A backfill is history, not state: a ref append that fills a new column
runs at its anchor, and a later capture of the table supersedes it. When a
step between the two reads the table or validates its rows, migrate over
that window refuses dml-superseded-in-window
(Preserved DML). The capture itself
still mints and exits 0, warning with the same code and naming the
backfill, the first such step and the recourse, and op seal refuses to
seal a range that would make the refusal unsplittable.
Text mode prints one captured … or already-captured … verdict per table.
JSON mode emits per-table rehearsal records on a dry run and closes every run
with an ok envelope whose data.tables array holds
the verdicts. --dry-run writes nothing and exits as the real run would.
Rows equal to the chain’s fold are already-captured and mint nothing, with
two exceptions:
- A
refchain at one anchor whosereplacefloor cannot render at that anchor (migratewould refusereplace-no-upsert-form, orkey-uncoveredunder an inbound foreign key) still mints the fresh floor. The verdict readssupersedes @<anchor> (no-upsert-form at the anchor)or(key-uncovered at the anchor)(supersedes_causein JSON). - When
import s.sqlhas moved a floor to anappend(see Re-ingesting into the corpus), capturings.sqlagain moves the key back even at identical rows. The capture mints the floor and reportsoverwrote @<anchor> (append → replace), orsupersedes @<anchor> (append → replace)once HEAD has moved (moved_from: "append"in JSON). Only the append’s own source moves it back: another source whose rows equal the append’s, a chain that still holds a floor, and a capture from stdin stayalready-captured.
A capture, or a data retire restate, at an anchor whose key holds only an
append overwrites that record, every source’s statements, when its rows
differ: the verdict reads overwrote @<anchor> and one overwrote: line
names each replaced file with its digest. A src=hand record at the key
refuses the run data-file-collision.
An unmatched --table pattern is an advisory, not a silent no-op.
A captured ref table that lost a key since its chain is still captured,
with a reference-key-lost warning naming the lost keys. A declared table
the source observes empty (its only statement an unfired TRUNCATE or
WHERE-less DELETE) mints replace body=none and warns
reference-table-emptied, --dry-run included; an already-captured
re-run, a seed chain and data retire do not warn. drift raises both
warnings for the tables it captures.
A <source> that is a stream (-, <(…), a FIFO) delivering zero bytes
refuses source-empty. A <source> that is a URL, a non-.sql
file or an extension-less regular file refuses kind-mismatch.
Each claimed table’s source band (the bytes of every INSERT statement
naming it, summed over the whole source) is bounded by the schema’s
data_band_max_bytes (see Create a project). A claimed table over the
bound refuses data-band-over-cap before any row is retained or
minted, --dry-run included; the envelope names every over-cap table with
its bytes, the bound and its statement count. A table --table filters out,
or one the run does not claim, is not counted. Recovery: raise
data_band_max_bytes for the named schema, or remove the key for the
built-in default.
retrofit data retire
data retire <schema.table> withdraws an existing data chain by minting
an empty restate floor at HEAD.
retrofit data retire public.countries
retrofit data retire public.accounts --label env:dev
retrofit data retire public.accounts --label base --label env:dev --dry-run
With no label, retire addresses the table’s unlabelled ref chain.
Repeated --label flags address the seed chain holding exactly that
label set, never a union of matching chains. If no such chain exists,
retire refuses no-such-chain and lists the chains that do.
A reference chain ends with its declaration. While the table is declared
an unlabelled retire refuses retire-declared-table, naming
schema ref-tables remove; once the declaration is removed it mints
nothing and reports the table not-managed, exit 0.
Removing a table’s declaration (schema ref-tables remove) ends its
reference span, and retrofit stops managing the rows. A fresh build
(snapshot at or past the removal, and data list) omits the records
anchored inside the ended span, cause undeclared (named under -v).
snapshot --at inside the span still renders them, and a migrate
window crossing the removal applies them as a delta against the fold at the
window’s start, by key, or by whole row where no key identifies one (see A window across a reference declaration’s
removal).
drift reports the chain not compared, information rather than drift.
The records and their witnesses stay as history. Declaring the table again
opens a new span; the old span’s records do not resume. A ref record
holding rows anchored on or past the removal refuses in every reader as
data-file-malformed, class ref-undeclared, and the body=none floor an
unlabelled retire minted before the removal is exempt; see data check
below.
Text verdicts say retired, already-retired or not-managed; JSON uses
the same rehearsal and terminal data.tables shapes as capture.
retrofit data scaffold
data scaffold mints a valid file beneath the resolved schema’s
data_root, at the deterministic path for its mint key, holding guidance
comments and one inert src=hand record: an INSERT over the table’s
live columns with a <column> placeholder per value. A --purpose cleanse
file holds a kind=DeleteStmt record instead, DELETE FROM <schema.table> WHERE <predicate>;, whose <predicate> is the placeholder to fill; its
guidance comments name the kind= tokens for a record rewritten as an
UPDATE:
retrofit data scaffold --table public.backfill --purpose transition --disposition append
retrofit data scaffold --table public.fixture --label env:test --disposition restate
Both --purpose values are experimental: without the project’s opt-in
the run refuses with experimental-not-enabled before it writes
anything, and with it the run warns experimental-purpose on the file it
writes (Experimental purposes).
Scaffold reads the table’s reference-data declaration at its minting cut,
as the other producers do. A declared table is minted purpose=ref, and
--purpose and --label are ignored for it; a run that passed either
raises a dml-ref-table-flag-ignored note naming the flag it ignored. The
run reports the minted file the same way for every band; the file’s header
line is what says purpose=ref:
dml-preserved: data/app.countries.202609281743.data.sql (declared-axes scaffold, 1 inert record to fill)
--purpose ref refuses with flag-invalid-value, since ref comes from
the declaration rather than a flag. With no declared purpose, no label,
and no policy match, the file is purpose=unclassified. --disposition
takes only append or restate; observed replace data belongs to
data capture.
--anchor-op overrides the reconciled head. A witnessed ref key refuses
(data-witnessed-key), and an anchor a seal covers refuses
(sealed-anchor) for every band but seed. A witness governs the ref
chain only, so a record of another band at the witnessed anchor is another
key and mints unrefused. A seed record may anchor inside a sealed range,
for a regression at the op where the bug lived; the mint raises a
seed-anchor-sealed note naming the seal that covers it, since a fresh
build and --at render the record there while a database already past
that seal never receives it.
A restate of a declared table states the whole table by hand and renders
as a replace; the mint raises a reference-restate-authored note (the
courtesy band: -q drops it), and data check raises it again over the
filled file.
data check is the fill gate: it refuses the file naming every
placeholder still present, and migrate and snapshot refuse an unfilled
scaffold (dml-body-unfilled) rather than reading it as empty.
Work with row data fills a scaffolded record and checks it.
retrofit data check
data check [path...] validates corpus files without emitting SQL or
minting anything.
retrofit data check
retrofit data check corpus/fixtures corpus/public.accounts.202609251400.data.sql
The command evaluates every addressed file and reports all findings together.
Malformed or conflicting files make the run fail; advisory-only findings such
as an unaddressed candidate filename do not change the exit status. A clean
text run reports data check: N file(s) checked, no findings; JSON emits a
data-check record with checked: N and an empty listed: {}. A run that
listed warnings but refused nothing counts them by code instead, as in
data check: 1 file(s) checked, no refusals; listed: anchor-shape-mismatch ×1,
and the JSON record has "listed":{"anchor-shape-mismatch":1}. The exit
is still 0, so a CI step that must stop on a listed warning gates on listed.
A record whose anchor-op= names no live op fails the run as
data-file-anchor-unresolved. Removing an op ends its id, and a revise
that changes an op’s content mints a new id for the new act, so either
leaves a record anchored on the old id; a reorder keeps the id, and the
anchor survives it. op revise, op remove, op unseal and a
single-file reconcile move the records under data_root that their
write leaves on a retired anchor (retrofit data carry), so the finding names a record that move
listed, or one the edit never saw, such as a file a merge brought in. Each
detail line names the record’s seq= and anchor, and the anchor’s fate
when the oplog still holds it: removed at a line, or replaced by a live
opid, the opid a re-pin may name. When a fate is named the hint names
retrofit data carry first; for one reviewed file,
retrofit data repin <file> --anchor-op <opid> re-pins every record in it.
A purpose=ref record holding rows whose anchor-op= resolves to a cut
where its table is not declared reference data fails the run as
data-file-malformed, class ref-undeclared, naming the file, the table
and the anchor. That covers a declaration withdrawn at or before the
anchor (schema ref-tables remove; the detail names the op that withdrew
it), a table never declared, and a table declared only after the record’s
anchor. The header’s table name is compared as the anchor spells it, so a
header naming the table by a later name is not declared there either.
snapshot, migrate and data list refuse the same file before emitting
anything. The recourse is not data repin: remove the file, and to keep
its rows declare the table again (schema ref-tables add) and capture
them at HEAD (data capture), which mints a record inside the new span.
A body=none floor is exempt, since it holds no rows; one anchored after
its table’s span ended (the floor an unlabelled retire minted before
schema ref-tables remove) is omitted as undeclared by every reader and
skipped by data witness. A project with no resolvable oplog yields no
such finding.
A purpose=ref restate body=none anchored where its table is
declared fails the run as data-file-malformed, class
ref-restate-declared, naming the file, the table and the anchor; no
producer mints this record (data retire refuses a declared table). The
recourse removes the file (rm <file>); to stop managing the table’s
rows, then end its span (schema ref-tables remove <table>), which, like
every corpus reader, refuses while the file stands. snapshot, migrate
and data list refuse the same file. The same file anchored where no span
is in force is the exempt floor above; a replace body=none is an
observation (see data capture) and a seed restate body=none is
data retire’s, and neither is refused.
A filled purpose=ref restate holding rows on a table declared at its
anchor passes, with the reference-restate-authored note data scaffold
raises (never as data-file-unfilled).
Two files at one key (the same table, band, label set and anchor) fail the
run as data-fork, whatever their dispositions: two floors, a floor and
an append, or two appends. snapshot, migrate and data list refuse
them the same way, under data_root and within each declared seed_paths
root (data check reads a seed root when it is named as an operand). A
merge that kept both sides, or a hand copy, is the usual cause: a
data capture branch and an import branch at one head reach it with no
hand editing. The refusal names the key and each file with its disposition
and source. The pages of one paginated capture are one record, two bands
or two label sets at one anchor are two keys, and a sibling schema’s files
under a shared data_root are left to that schema’s own check.
The data-fork hint names one of two recourses:
- Re-run the delta’s import. For a floor and a delta read from a
source file, the hint names removing the delta’s file and re-running its
import, which adds the delta as a section of the floor’s record. The re-run repeats the schema the check read (a typed--schemaas typed, a schema chosen with-fby its registered name), a typed-Cand--config, a typed--data-root(as--seed-pathfor a labelled delta), and the source relative to the working directory. A delta whose import took--disposition appendneeds it again. - Keep one record. Two floors, a delta with no source to re-run
(
src=hand, or read from a stream), a floor holdingsrc=handwork or no records, a delta from the floor’s own source, and a re-run that would not land on the floor keep one record by removing the others’ files; a paginated record keeps all its pages. A re-run misses the floor at a key anchored before the oplog’s head (an import anchors at the head), at a floor outside the root it writes to (a labelled delta underdata_rootwithseed_pathsdeclared and no--data-roottyped lands under the first seed path), and on a run with no registered schema to import into.
Beside other findings the fork joins the same single envelope, with its own recourse.
Two cases raise no anchor finding. A project with no oplog skips the check,
since import-driven flows write data files before any op exists. A record that
belongs to a sibling schema under a shared data_root is left to that
schema’s own data check.
A corpus holding a transition or cleanse record the project has not
opted into refuses the run as experimental-not-enabled before any
record is read, whichever files the operands name
(Experimental purposes).
Refusal classes
A refused file’s line leads with its class, a name for what is wrong with
the file (data/c.data.sql: writes-other-table record seq=1 writes relation "b" …), and the text refusal repeats the leading one as class:. In JSON
the class is the discriminator of each entry under files, and a
data-file-malformed envelope’s own discriminator names the most specific
one, so a script branches on the name rather than the prose. A file with
several faults is one entry, its detail naming each class.
Every corpus reader (data check, snapshot, migrate, data list)
refuses these as data-file-malformed, and a producer refuses to write a
file that would have one, except where a row names the one side that
checks it:
| class | what it catches |
|---|---|
grammar | The header line or the record fences break the data-file grammar: a missing, duplicated, unrecognized or out-of-order header token, a value outside its set, more than one plain record fence, or a delta section where none may stand. |
unknown-form | body= or format= names no form this version admits. |
manifest-reserved | body=manifest, which is reserved and not yet admitted. |
fence-on-body-none | A body=none file holding a record fence. |
format-on-body-none | A body=none file holding format=. |
append-body-none | disposition=append with body=none: an append adds rows, so it cannot declare a table empty. |
unfilled | A record still holding data scaffold’s <column> placeholders. data scaffold writes them; data check refuses the file, migrate and snapshot refuse it as dml-body-unfilled, and data list lists it. An append whose fence holds no records only warns data-file-unfilled. |
unfilled-floor | A floor (replace or restate) with body=data whose own section holds no records. data scaffold writes this shape; it is refused when read. |
writes-other-table | A record writing a table other than the header’s table=, or more than one table. |
replace-kind | A replace record that is not an INSERT or a COPY supplying rows. |
extension-mismatch | The file name’s extension disagrees with format=. |
table-not-live | table= names no table live at the anchor. Checked when a producer writes the file and when data repin moves it. |
purpose-outside-band | A floor whose purpose= is outside the band the fold admits. |
ref-undeclared | A purpose=ref record holding rows, anchored where its table is not declared reference data (above). |
ref-restate-declared | A purpose=ref restate body=none anchored where its table is declared (above). |
A framing fault is one of the framing classes, also
as data-file-malformed.
A file the reader cannot open is listed under files with class
unreadable; when every refused file is unreadable, the run refuses
data-file-unreadable instead, an envelope with no discriminator of its
own.
A producer’s own refusals name a class too:
| code | class | what it catches |
|---|---|---|
data-file-malformed | cascade | An imported TRUNCATE … CASCADE: import cannot see which tables the cascade reaches, so it mints no floor for them. |
data-file-malformed | interleaved-statement | An imported statement on a table between the statement that empties it and the INSERTs that refill it. |
data-file-malformed | off-grain | A file a producer would remove as superseded that holds more than one table, label set or anchor, or no longer parses; it is refused rather than partly edited. |
data-file-collision | key-collision | A write meeting a file it may not replace at its key: another key’s or another purpose’s record, hand-authored records, a file it cannot read, two pages landing on one path, or a floor no delta joins. |
purpose-disposition-conflict | purpose-flag-outside-band | A --purpose outside the fold’s band on a run that would write a floor. |
A fault between files keeps its own code when no file is malformed, and its
envelope then has no discriminator: page-set-mismatch and
anchor-unresolved still list each file under files with its class, while
seq-claimed alone names the files only in its detail, so a script branches
on the code. Alongside a malformed file, each joins the data-file-malformed
envelope, every file’s entry under its own class:
| code | class | what it catches |
|---|---|---|
data-file-order-ambiguous | seq-claimed | Two files at one anchor claim the same seq=. |
data-file-page-set | page-set-mismatch | A paginated record whose pages are not exactly 1 to N: a page missing, repeated or out of range, or pages disagreeing on N. |
data-file-anchor-unresolved | anchor-unresolved | A record whose anchor-op= names no live op (above). |
Framing classes
A record is one marker pair, and its body may hold one statement, several
statements or one multi-row INSERT. Both marker lines are required, and
every way to get the framing wrong refuses by name rather than
mis-parsing:
| class | what it catches | remedy |
|---|---|---|
end-missing | A record’s declared end line never arrives before EOF. | Close the record with its declared end line. |
marker-in-body | A marker-prefixed line inside a record body that was never closed: the next record’s marker read as body content. | Close the first record with -- retrofit:dml-end, or escalate the frame: add end=<tag> to its marker line and close the body with -- retrofit:dml-end:<tag>. |
end-line-whitespace | A line that differs from the declared end line only by trailing whitespace or a carriage return: an end line the scan cannot match. | Normalize the file’s line endings; the body is otherwise correct. |
unframed-content | Content between records that no marker pair frames, typically statements pasted straight into the section. | Wrap the statements in a marker pair. |
retired-bytes-token | A marker line holding a bytes= token. | Drop the token, and close the body with its end line. |
bad-marker-token | A marker line holding an unrecognized, duplicated or mis-ordered token, or a malformed end= tag. | Correct the token to the record marker grammar. |
duplicate-end-tag | The same end= tag claimed by two records in one file. | Give the record a tag free across the whole file; the refusal names the lowest free tag. |
byte-order-mark | A UTF-8 byte-order mark at the start of the file. | Strip the BOM. |
end-past-section | A tagged record’s end line never arrives before the section end; the scan crossed the fence looking for it. | Close the record inside its section. |
empty-body | A record whose body is empty: its marker line is immediately followed by its end line. | Give the record a body, or remove the record. |
section-end-in-minted-body | Write-path only. A body holding a -- retrofit:dml-section-end line that would be minted inside a section, where a tagged scan would swallow the fence’s terminator. | Place the record outside a section. |
section-end-in-body | The read-path mirror of section-end-in-minted-body: a record whose body consumed the enclosing section’s end line. | Place the record outside a section. |
duplicate-seq | The same seq= claimed twice within one file. | Renumber so each record in the file claims a distinct seq=. |
Every class but section-end-in-minted-body is a data check refusal as
data-file-malformed, with the name as the refusal’s class and the
offending line, so the refusal names the fix. section-end-in-minted-body
is checked before any byte is minted, over the whole set at once, so a
refused mint leaves no partial file behind; section-end-in-body covers a
file that acquired that shape some other way.
A reference append (or restate) whose body is an UPDATE, DELETE, MERGE,
INSERT … SELECT, or an INSERT leaving a column DEFAULT is legal and
applies once in its window, but the chain no longer folds offline: drift
cannot compare it and removed keys cannot be computed until the next version.
data witness still witnesses the record and warns
reference-chain-unfoldable naming the table, the record’s anchor and the
statement that does not fold (structured: table, opid, reason); the exit
is unchanged. A literal-INSERT append folds and raises nothing.
retrofit data witness
data witness [path...] binds reference records’ bytes to the oplog: it
appends one WitnessReferenceData entry per selected record, over every
file at its key. validate, op seal, migrate and snapshot verify a
witnessed record whose anchor a seal covers.
retrofit data witness # every reference record in the unsealed tail without a witness at its bytes
retrofit data witness --dry-run # the same rows, appending nothing
retrofit data witness corpus/public.colors.202609251400.data.sql
A directory operand refuses with witness-operand-directory. A named
record outside the selection refuses with witness-band (a file that is
not a reference record under data_root, or a record anchored outside its
table’s reference span) or sealed-anchor (a seal covers its anchor). One
already witnessed at its current bytes is reported already-witnessed. A
data_root holding a transition or cleanse record the project has not
opted into refuses the run as experimental-not-enabled
(Experimental purposes).
retrofit data repin
data repin <file> --anchor-op <target> re-points one corpus data file’s
anchor-op= at an opid you name: the reviewed recourse of a record whose
anchor names no live op (data-file-anchor-unresolved, above), or whose
row shape moved at an anchor that still resolves (anchor-shape-mismatch).
retrofit data repin corpus/public.colors.202609251400.data.sql --anchor-op last --dry-run
retrofit data repin corpus/public.colors.202609251400.data.sql --anchor-op 4c1382aed592
The target takes migrate --to’s grammar, and what is written is always
the resolved 12-hex opid. Without --anchor-op the verb refuses
anchor-target-required. Run retrofit op show to read the opids.
A file is eligible when its anchor names no live op, or when its anchor
resolves but the row shape there no longer matches its anchor-shape=
token (or the token names a version or algorithm this binary cannot read).
A file whose anchor resolves with a matching shape, or with no token,
refuses anchor-still-resolves; capture the rows at the new point with
data capture instead. A moved shape at an anchor a seal covers refuses
sealed-anchor, its hint naming op unseal first. A file already at the
target is already-pinned, exits 0 and writes nothing when its shape
matches; when the shape moved, it is re-stamped at that anchor and
reported restamped.
A re-pin rewrites the header’s anchor-op=, writes the anchor-shape=
token at the target when the body has a shape, and re-stamps each
record’s seq= after the records already at the target, as a mint there
would. Every other byte of the file (body rows, comments, record markers
and the other header tokens) is passed through unaltered. --dry-run
computes and reports the same decision, prefixed would repin: , and
writes nothing.
A re-pin of a witnessed reference record re-witnesses it: the verb appends
one WitnessReferenceData entry at the new anchor, over the rewritten
bytes, and names its opid. The old witness is left in place. An
unwitnessed record stays unwitnessed, and --dry-run appends nothing.
The target is also checked for legality before any byte is written, and each of these refuses with the file unchanged:
| Target | Refusal |
|---|---|
The cut before the first op (root). | anchor-target-pre-genesis |
| A cut where the file’s table is not live. | data-file-malformed, class table-not-live |
A cut where a ref record’s table is not declared reference data. A body=none floor is exempt, since it holds no rows. | anchor-target-undeclared |
A cut where a ref restate body=none floor’s table is declared. | data-file-malformed, class ref-restate-declared, the refusal data check would give the moved file |
| A cut another file already claims for the same table and labels. | data-file-collision |
A cut a seal covers, for a record that is not purpose=seed. | sealed-anchor |
A paginated file refuses with data-file-paginated before any of these
unless --anchor-op is the page’s own full 12-hex anchor-op=, which
re-stamps that page where it stands. The verb does not move a page set,
so re-capture it at HEAD instead. Between that check and the target’s, a
transition or cleanse file, or a data_root holding one, refuses with
experimental-not-enabled unless the project opted into its purpose
(Experimental purposes).
A seed record re-pins into a sealed range, and the run names the seal that
covers the target (seed-anchor-sealed).
retrofit data carry
data carry [<file> ...] moves records under data_root onto the anchor
that now holds their place, and named files narrow the sweep to those
records. op revise, op remove, op unseal and a single-file
reconcile run the same sweep after their own write and print its report.
retrofit data carry --dry-run
retrofit data carry
Two kinds of record move:
- A retired anchor. A record of any band whose
anchor-op=the oplog revised or removed moves to the replacing op, or to the op before a removed line, followed until it names a live op. An oplog in the compact form keeps no trace of a retired op, so there the record is listed fordata repin; one on a removed first line, which has no op before it, is listed for a capture at HEAD. - A release behind it. A reference record whose anchor a seal covers, and which has no witness in that seal’s range, moves onto the newest Seal line. A seed record is never moved this way, and a witness whose digest no longer matches is an integrity finding, never moved.
A record moves only when its anchor-shape= equals the row shape at the
target; a mismatch, or no token, is listed. A move rewrites anchor-op=
and re-stamps seq= as a mint at the target would. Each witness of the
record moves with it, its digest kept, and is stale until
retrofit data witness; the report then ends with a note naming
data witness before op seal. Run alone or by reconcile, a record
whose target key another record already holds is listed as a fork, never
merged.
The report is one line per record, in file-path order:
carried: <file>: <table>@<from> → @<to> (<cause>): <clause>
listed: <file>: <table>@<from>: <reason>; <recourse>
fork: <file>: <table>@<from>: <reason>; <recourse>
not selected: <file>: <table>@<from>: <reason>
<cause> is revised, removed or release <name>. not selected: is
a named file the sweep does not pick. Listed records exit 0. --dry-run
prints would carry: for carried:, every other line unchanged, and
writes nothing; a sweep with no line prints nothing to carry. Under
--as=json each line is one record with file, from, to, cause,
outcome (carried, listed, fork or not-selected), reason and
recourse on a record that did not move, holder on a fork, and dry_run: true on a
rehearsal.
A named path that is not a readable record file refuses
data-file-unreadable, or data-file-malformed for a readable file that
is not a record, before any write. When the sweep fails after a
calling verb’s write, the write stands and the run exits 1 naming
retrofit data carry; a rerun completes it.
retrofit schema ref-tables
ref-tables declares which tables hold reference data (rows the
application reads but never writes, such as currency codes, country lists and
status enumerations), applied on every emit with no --label. It is the
first step of the classification chain in Purpose, and the
declared set is the reference capture scope every producer run reads at its
own minting cut: import, data capture, drift (which captures the
declared tables at its new head on every run, with no flag), and
data scaffold.
add and remove append one SetReferenceData op per resolved table to
the oplog. A table can also be declared as it is created
(op add CreateTable … reference_data=true, or --ref <table> on import
and drift), and ref-tables reads those declarations like its own.
retrofit schema ref-tables # read the declared set and report
retrofit schema ref-tables add countries 'curr*' # declare the tables the arguments resolve to, then report
retrofit schema ref-tables remove currencies # withdraw a declaration, then report
All three forms print the same report, the whole declared set rather than a delta, so a read and an edit compare byte for byte.
Each argument is an exact table name or a glob, resolved against the tables
live at the read cut: HEAD, unless --at <opid> names another. A glob
expands when you run the command, so a table created later is not declared
by an earlier glob, and a glob never declares an attached partition child:
it reports each one as skipped: instead. An argument that resolves to
nothing is refused (ref-table-not-live): import the DDL, drift it in, or
create it with reference_data=true, then declare it. Declaring a table
that is already declared, or removing one that is not, is a per-name no-op
that mints nothing and says so (already-declared, not-declared) rather
than refusing; --ref and op add SetReferenceData answer the same way.
The exception is a merge or join product whose sources disagreed about the
declaration: op seal refuses it until a SetReferenceData on it records
the judgment, so every one of those writes mints that line even though the
product already reads declared.
The report lists the declared set as one declared: line per table, in
bytewise order. On add and remove one line per resolved table follows: the
table name, then the SetReferenceData op the run minted (or, under
--dry-run, would mint) as its on-disk oplog entry verbatim,
SetReferenceData(<opid>, prev=<prev>, on=<element-id>, value=<bool>), the
same line that lands in the oplog. It is a record of what was written, not a
command to run. Then comes one line per disagreeing table, and a run-level
summary:
$ retrofit schema ref-tables add countries 'curr*' --schema app
declared: countries
declared: currencies
countries SetReferenceData(00664d8eaeaa, prev=6ccda870b705, on=081f8bafa7ea, value=true)
currencies SetReferenceData(32bbc6fd0f0e, prev=00664d8eaeaa, on=2513d190b2ad, value=true)
ok app declared_tables=2 disagreeing_tables=0 dry_run=false
$ retrofit schema ref-tables add countries loan_periods --dry-run --schema app
declared: countries
declared: currencies
declared: loan_periods
countries already-declared
loan_periods SetReferenceData(8c78aa7cd7e2, prev=32bbc6fd0f0e, on=49448cf0ecb0, value=true)
ok app declared_tables=3 disagreeing_tables=0 dry_run=true
A run with --at appends at=<opid> to the summary. The full grammar,
text and JSON, is the schema ref-tables row of
Output contract. --data-root
moves the base corpus the disagreement report reads; it adds no seed
layer.
The first declaration in a project whose data_root is the undeclared
default and does not exist yet creates that root with an empty .gitkeep,
and says so on stderr, so every checkout has the root the declaration’s
records will land in:
created data/.gitkeep so every checkout has the data root; keep it with the reference-table declaration
Commit the file with the declaration. A root you named (the data_root key or
--data-root) is never created.
Refusals
Argument failures accumulate: every argument is evaluated and each
failure is reported beside the argument that caused it, rather than the run
stopping at the first. The envelope has the covering code
ref-tables-rejected, each entry in its violations array has its own
per-failure code (see Output contract), and the
whole invocation is refused, so nothing is minted, not even for the
arguments that resolved.
| Code | When |
|---|---|
ref-table-not-live | The argument resolves to no table live at the read cut: an exact name no live table has, a glob that matches none, or a glob whose every match is an attached partition child. |
ref-table-pattern-invalid | The argument is not a well-formed glob (an unterminated character class, say). |
ref-table-pattern-qualified | The argument qualifies itself with the selected schema’s own name. The schema is already selected (by --schema, or from the config chain when it is omitted), so the argument takes a bare name. |
reference-data-partition | The argument names an attached partition child exactly. The refusal names its partition root; declare the root instead. |
ref-tables-rejected | The covering envelope code holding the array above. It names the envelope’s kind, never the first failing argument’s. |
ref-tables-schema-not-in-cut | The named schema is registered in retrofit.toml but absent from the oplog at the read cut, so there is no live table universe to resolve against. Distinct from unregistered-schema, which is the registry miss. |
An unregistered schema refuses with unregistered-schema. Each code’s exit
status is on Refusal codes.
Notes
Three advisories follow the report at the courtesy band (they drop under
-q): dml-ref-table-no-match-universe (no table is live at HEAD, so an
argument is neither declared nor refused), dml-ref-table-policy-disagreement
and dml-ref-table-seed-dormant (Advisory codes). An
add (and add --dry-run) that moves a table holding a seed chain into a
reference span, in the base corpus or a seed_paths root, raises one
dormant note per chain, naming its label set and earliest anchor; its hint
names the chain’s recourse, as listed under data list above. op add SetReferenceData <table> true and import/drift --ref on a live table
raise the same note.
The data file
Custom DML (INSERT, UPDATE, DELETE and MERGE) is outside
retrofit’s vocabulary, so no op records it. import writes each such
statement, verbatim, to a runnable file per table instead: the data
file. data capture, drift and data scaffold write the same
files. The corpus is kept outside the oplog (Data
files explains how it follows the schema). snapshot
and migrate read every addressed *.data.<format> file under
data_root on every run
(The data band,
Preserved DML).
A run that writes custom DML writes one data file per table:
- Name:
<schema>.<table>.<YYYYMMDDHHMM>[.p<K>].data.sql, the table and then the UTC minute the run captured it at. The stamp is opaque and never read back. - Location: the schema’s
data_root, a[[schema]]field inretrofit.tomlthatschema createscaffolds; unset, it is[project] data_root, thendataunder the project root. A labeled record is in a directory named after its first label in sorted order (data/<label>/…). A label, schema or table name that is not only lowercase letters, digits and_is written folded to those characters with a short digest after a-, so the labelseed:xis the directoryseedx-<digest>. - Seed roots:
seed_paths(in[project], or in a[[schema]], where it replaces the project’s list; a[[schema]]’sextra_seed_pathsadds to it) names further roots holdingseedrecords, and aseedrecord is written under the first entry when one is declared.--data-root,--seed-pathand--extra-seed-pathreplacedata_root, replaceseed_pathsand add a seed root for one run. - Key:
{table, band, label-set, anchor}, stated in the header. A record of another band (ref,seed, or thehistoryband thattransitionandcleanseshare) at the same anchor is another key and is left standing, so animport --purpose transitionbeside a reference record is a second file, never a rewrite. - Re-capture at a held key: a floor is overwritten, an
appendaccumulates (at a floor’s key, as the floor’sdeltasection), the floor’s own source moves it, a witnessed key refusesdata-witnessed-key, and identical records write nothing. See Re-ingesting into the corpus. - Name collision: a computed name already held by a record of a different key takes the next minute, and the next, until the name is free, so two records of one table written in one minute are two files.
- Readers use the header: every fact comes from the header, never the name, so a moved or renamed file still resolves.
One table per file is the grain to curate by: rm data/<label>/c.*
removes table c at every anchor. Two shapes hold more than one file
for a table: a capture whose record count exceeds the page bound is a
page set (.p2, .p3, … with page=K/N on each header), and a
floor that a later append at its anchor does not join (a hand-authored
floor, or a scaffold’s empty floor) stands beside that append’s own
file. Removing one member of either does not curate the table out (the
sibling still speaks for it), so curate the set, and let retrofit data check confirm the corpus reads cleanly before you commit.
No producer runs a data file. The rows reach a database only when you
apply them: run the file yourself, or apply the SQL snapshot or
migrate emits, which holds the bodies.
File layout
Each file is line-oriented. A v1 header tuple names the source,
the one table the file declares, the anchor it is pinned at, and the
file’s axes (purpose, disposition, body, format, labels, and
page on a paginated capture). Every record is inside the file’s
single body fence: a marker line, its verbatim text, and an end
line that closes the body. The body is written exactly as captured,
with nothing appended: a statement that arrives without a trailing ;
is stored without one, and migrate supplies the terminator when it
renders:
-- retrofit:data-file v1 source="01_seed.sql" table=["public","users"] anchor-op=988edcb612ab purpose=seed disposition=append body=data format=sql labels="[\"env:dev\"]"
-- retrofit:dml-section
-- retrofit:dml seq=1 src-file="01_seed.sql" src-line=42 kind=InsertStmt
INSERT INTO users (id, email) VALUES (1, 'a@example.com');
-- retrofit:dml-end
-- retrofit:dml seq=2 src-file="01_seed.sql" src-line=44 kind=UpdateStmt
UPDATE users SET active = true WHERE id = 1;
-- retrofit:dml-end
-- retrofit:dml-section-end
A floor’s file may hold a second, delta section after the floor’s
own fence: a delta that meets the floor at its key (an import after a
data capture, or a script’s UPDATE after its TRUNCATE; INSERT)
joins the floor’s record rather than starting a second file. Its fence
opens -- retrofit:dml-section delta, the one token a fence admits;
the header keeps disposition=replace, its records are deltas (any
DML, each naming its own src-file), and migrate loads the floor,
then runs them:
-- retrofit:data-file v1 source="fixtures/track.sql" table=["app","track"] anchor-op=0123456789ab purpose=seed disposition=replace body=data format=sql labels="[\"qa\"]"
-- retrofit:dml-section
-- retrofit:dml seq=1 src-file="fixtures/track.sql" src-line=1 kind=InsertStmt
INSERT INTO app.track (id, user_id, status) VALUES (1, 42, 'open');
-- retrofit:dml-end
-- retrofit:dml-section-end
-- retrofit:dml-section delta
-- retrofit:dml seq=2 src-file="refund-case.sql" src-line=1 kind=UpdateStmt
UPDATE app.track SET status = 'refund_failed' WHERE id = 1;
-- retrofit:dml-end
-- retrofit:dml-section-end
A body=none file is a floor (it declares the table empty at its
anchor) and has no fence at all: the header is the whole file. A delta
that meets an empty floor at its key (TRUNCATE t;, then an INSERT,
in one script or a later import) joins it the same way: the record is
rewritten body=data with an empty floor fence, then the delta
section, and migrate clears the table, then runs the delta.
The body is the whole lines between the marker line and the end line. The newline just before the end line is part of the framing, not of the body, so a body never gains a trailing newline its author did not write. Everything in between is kept byte for byte: no escaping, no unescaping, no normalization.
Header fields
The retrofit:data-file and retrofit:dml marker tokens are the
on-disk grammar and stay stable for re-ingest. The header tuple states
the file’s facts in this fixed order:
source: where the file came from (the import source, a drift dump, orscaffold). Provenance only; nothing checks it.table: the one table the file speaks for, as a two-element JSON array["<schema>","<relname>"], each half spelled exactly as the oplog spells it. Every record body must write that table and no other (classwrites-other-table); a record that writes another relation belongs in its own file.anchor-op: the 12-hex opid of the op appended immediately before the statements’ source position.migratesplices the bodies back in just after that op’s DDL. A statement with no DDL before it in its source file falls back to that run’s first op; a data-only import (no DDL) anchors at the oplog’s last op instead. An authored statement whose table does not stand at its anchor refuses withdml-table-not-found, and an empty oplog models no table, so a data-only import that captures a statement into one refuses that way (a statement no step of the purpose chain classifies is listed, not captured). Statements anchored at different ops are in different files.anchor-shape: a digest (v1:sha256:<hex>) of the columns the record writes, as they stand at the anchor.data checklists a record whose digest no longer matches (anchor-shape-mismatch), andsnapshotandmigraterefuse it;data repinre-stamps it.purpose: the classification (see Purpose). Absent meansunclassified.disposition:append,replaceorrestate.replacedeclares the file is the table’s full observed row set at its anchor; import infers it from aTRUNCATEor unqualifiedDELETE: the INSERTs for that table that follow the arming statement are written as onereplacefile, and an arming statement with no following INSERT writes areplace body=nonefloor. Everything else isappend;import --dispositionsets it for the whole run.- A statement the source states before the arming statement, at
the floor’s own anchor, is wiped by it: the floor supersedes it,
nothing is written for it, and the run names it
dml-superseded: <src-file>:<line> (<schema.table>; superseded by the arming statement at <src-file>:<line>)({"type":"dml-superseded","table":…,"src_file":…,"src_line":N,"arming_file":…,"arming_line":N}). A statement at another anchor (DDL between) keeps its own record, and one that also writes another table (a data-modifying CTE) is never superseded: the write refuses its second relation instead. - A non-INSERT on the table stated after the arming statement but
before or among the INSERTs that follow it, at the floor’s anchor,
is neither a floor row nor a delta that follows the floor, so the
import refuses before writing anything (dry run included), naming
the arming statement, that statement and the first INSERT after it.
Replace the arming statement with a qualified
DELETE FROM t WHERE true;(which never arms the window) so the script is a delta, or move the INSERTs ahead of the statement; one stated after the last INSERT is the floor’s delta section. restate(hand-authored or scaffolded only) restates the table’s rows for exactly the file’s label set, ignoring that chain’s earlier records.- A
replaceorrestatefile must have a fold-band purpose (seedorref). Records of one (table, label-set) chain are versioned by their anchor-op position: the latest floor at or before a render point supersedes earlier state, whilesnapshot --atstill renders older versions.
- A statement the source states before the arming statement, at
the floor’s own anchor, is wiped by it: the floor supersedes it,
nothing is written for it, and the run names it
body:data(the fence holds the rows) ornone(a floor declaring the table empty; no fence). Absent meansdata.format:sql, the inline body’s serialization; written on abody=datafile only, and the file’s extension must agree with it (.data.sql).labels: the optional label set (see Labels), written only when the file has at least one label.page:K/Non a paginated capture only.
Record markers
Each record marker inside the fence has:
seq: a 1-based counter in capture order, one stream across every file a run writes, so records merge back into source order across files. Two records at one anchor with the sameseqin different files are an order ambiguity, refused withdata-file-order-ambiguous.src-file(relative to the project root) andsrc-line: where the statement came from, JSON-quoted likesource. A hand-authored record hassrc=handinstead.kind: the statement’s parse-tree node kind (InsertStmt, …). Areplacefile admitsInsertStmt(and hand-authoredCopyStmt) records only.end: the optional framing tag (see Escalated framing). Present only when the body needs one; it names the end line that closes this record, and it is the marker’s last token.
Escalated framing
The default end line is the fixed text -- retrofit:dml-end. That
works until a body itself contains a line starting with
-- retrofit:dml, at which point a reader could not tell the body’s
line from the real frame. So:
A record must use escalated framing whenever any line of its body
begins, at column 0, with the literal -- retrofit:dml. Escalated
framing adds an end=<tag> token to the marker line and closes the
body with -- retrofit:dml-end:<tag> instead:
-- retrofit:dml seq=3 src-file="03_notes.sql" src-line=7 kind=InsertStmt end=1
INSERT INTO notes VALUES ('a literal
-- retrofit:dml-end
inside a string, so this record escalated its tag');
-- retrofit:dml-end:1
A tagged record is closed only by its own tagged end line. Inside
one, every other marker-like line (including
-- retrofit:dml-section-end) is ordinary body text, so escalated
framing is how a body holds lines that look like markers.
Choosing a tag:
- A tag is a decimal integer with no leading zeros (
1,2,17). - Tags are unique across the whole file, not per record and not
per section. The tag assigned is the lowest one still free in the
file, in record order, so the first escalating record usually takes
1, the second2, and so on. A record that does not escalate uses no tag, which is why a file may holdend=1, then an untagged record, thenend=2. - A tag must also be absent from the body it closes: if the body
already contains the line
-- retrofit:dml-end:<tag>at column 0, that tag would close the record early, so it is skipped. Tag allocation may therefore skip integers: a body containing-- retrofit:dml-end:1getsend=2even when1is otherwise free. - When you add or copy a record by hand, pick a tag that is both free
across the entire file and absent from that record’s body. Reusing
one fails the next read with
data-file-malformed, and the message names the lowest tag still available across the file; if that tag collides with a line in your body, take the next one that does not.
The literal prefix is -- retrofit:dml and nothing wider: the
-- retrofit:raw, -- retrofit:snapshot-data and -- retrofit:data
markers belong to other features and are ordinary body content, so
they never force escalation.
One placement rule follows: a record whose body contains a column-0
-- retrofit:dml-section-end line cannot be placed inside a
section, because its body would swallow the section’s own terminator.
That shape fails with data-file-malformed on write and on read. The
same body outside a section is fine.
If a data file will not parse
A framing fault fails the read with data-file-malformed, naming the
record and line under a class that names the fault, such as
end-line-whitespace or marker-in-body (every
class). The common ones:
- A body’s end line is missing, or was damaged; a CRLF rewrite is the usual cause, since end-line recognition is byte-exact. When only trailing whitespace differs, the message says so, because the fix is your editor’s line-ending setting rather than the body.
- A body contains a
-- retrofit:dmlline but the record was not escalated. The remedy is the tag escalation above. - Anything other than blank lines and ordinary
--comments sits between one record’s end line and the next marker line. - A record’s body is empty.
- A marker line has a
bytes=token. Delete it from each marker line and close each body with a-- retrofit:dml-endline.
Purpose
Each record has a purpose, which tells the emit commands what to do
with it; What a data file is for
has a table of the five. Beside that table:
transitionandcleanseare experimental: a project uses them only by opting in (Experimental purposes).cleanseis emitted under arun-oncemarker.refis never label-filtered.- A
seedrecord no--labelnames is excluded and counted, causenot-selected. unclassifiedis the fail-closed default. A header without apurpose=token reads as it, anddata scaffoldwrites it when no purpose, label or policy match applies;importsets aside a statement it cannot classify rather than write it.migraterefuses an unclassified body unless you pass--allow-unclassified-dml;snapshotexcludes and counts it.
import classifies each statement it preserves once; the first step
that applies decides. data capture and
data scaffold classify by their own rules.
- The reference-data declaration, an attribute in the oplog, not
configuration. A literal
INSERTthe reference fold admits (anON CONFLICTclause, for one, fails it) into a table whose read at the run’s cut isreference_data=trueisref, whatever--purposeor--labelsays. --purpose transitionor--purpose cleanse: a run-level assertion stamped onto every other statement the run preserves.--purposetakes no other value, and is not combined with--label. Either value needs the project’s opt-in (Experimental purposes); without it the run refuses before it writes anything.- A declared table, otherwise: any other statement into a declared
reference table is listed with cause
no rung; it is neverseedand neverref. --label: a run that names one or more labels writes its records asseed, with the labels stamped on them. A label names a dataset, and a dataset is a seed band, so the band follows from the label.- Otherwise, the statement is listed and written to no data file:
an authored statement with cause
no purpose or label, a dump’s data band with causenot declared. There is no defaulttransitionorseedstep: a purpose must be asserted, target a declared reference table, or follow from--label.
Experimental purposes
transition and cleanse are experimental (Data
files says why). A project uses
either purpose only by opting in, in retrofit.toml:
[experimental]
purposes = ["transition", "cleanse"]
The table is project-wide, and the opt-in is per purpose: a project that
lists only transition still refuses a cleanse record. A name other
than the two, a name listed twice, a purposes that is not an array of
strings, or any other key in the table refuses at load as
config-invalid.
Without the opt-in, both purposes refuse as experimental-not-enabled,
before anything is written and under --dry-run too:
import --purposeanddata scaffold --purposerefuse to write a record of either purpose.validate,import,data scaffold,data list,data check,data witness,data repin,schema ref-tables,migrateandsnapshotrefuse a data corpus holding one, before any record is read. The check reads every data file’s header, whatever the run applies, sodata check <file>andsnapshot --ddl-onlyrefuse too. The detail names each file and its purpose, one<file>: purpose=<purpose>line per file.
The refusal’s hint names the key and the alternative below. With the
opt-in, each verb reads and writes these records as it does any other,
and prints an experimental-purpose warning once per such file it
reads or writes. The warning does not change the exit.
The alternative is a backfill written as a DO block. import and op add record it as a RunRawScript line in the oplog, which a Seal’s
digest covers. Declare its reversal on the same line; here
reverse_noop says a down needs no action:
retrofit op add RunRawScript \
'DO $$ BEGIN UPDATE public.orders SET total = 0 WHERE total IS NULL; END $$;' \
'reverse_noop=the backfilled totals stay after a down'
With no reversal declared, a down over the script refuses, and import
and op add report reverse-undeclared when they record it
(The reverse migration).
Labels
--label <label> (repeatable) names a dataset for the run: step 4 of
the purpose chain. Labels select cohorts within the seed
band, so only seed records have them: a record step 1 or 2 classified
ref, transition or cleanse, or that the chain lists, is never
labeled. The value is free-form (a key:value form is accepted, not
required); labels are sorted and deduplicated on disk. They mean nothing
to retrofit; they let you name the dataset at emit time with
--label <label> on snapshot or migrate.
Re-ingesting into the corpus
A re-ingest whose records anchor at a new op writes a new data file. At a key that already holds a record:
- Identical records write nothing.
- A divergent floor (
disposition=replaceorrestate) at an unwitnessed key replaces that key’s version as a set: matched files are rewritten in place under their original names, surplus pages are deleted and named in the report, missing pages are written, and each replaced file is reportedoverwrote:with its digest. Only a commit keeps the prior bytes. - A floor at an
append’s key (--disposition replacewhere no floor stands) overwrites the standing record, every source’s statements, when its rows differ; another source’s equal rows leave the record as it stands. Each replaced file is reportedoverwrote:with its digest. Asrc=handrecord there refusesdata-file-collision. - An
appendaccumulates: the incoming statements join the standing record after its own, reporteddata-appended:with the digest and the statements added. At an unwitnessed floor’s key they join the floor as itsdeltasection, reporteddata-appended: … (replace; delta <src> added, +N statements); an identical floor re-capture leaves the record as it stands, and a divergent one replaces it whole. Two data-only imports into one table add no op, so they share the anchor and leave both sources’ statements in the one record, in import order. - Accumulation is per source file. Re-importing an unchanged
src-filechanges nothing, and re-importing it changed replaces that source’s statements where they stand rather than adding them again, so it still replays before the sources imported after it. A changed source that now holds more statements than before keeps its place too: the extra statements take the next seqs after its section, and the record’s later statements shift up by the growth, in order, reportedN shifted (<src-file>). Seqs an earlier shrink freed are room. When the shift would reach a seq another file holds at the anchor (another label set’s record, say), the import refusesdata-file-collisionnaming that file and seq before anything is written: another key’s file is never re-stamped, and the section is never moved to the end. - Statements across sources. A changed source’s old statements go
even when every new one is a duplicate another source holds. A
statement restating a keyed row another source states supersedes that
source’s statements. The header’s
source=stays the first writer’s, each statement keeps its ownsrc-fileandsrc-line, and a statement another source already holds is skipped as a duplicate. - An
appendof a different purpose than the standing record’s is not merged into it and overwrites it. - A witnessed key refuses
data-witnessed-key.
Importing the source a floor was captured from (data capture s.sql,
then import s.sql) moves the floor rather than deduplicating against
it. The last command run on a source decides what the key says about
it, whatever the bytes, so the floor’s file is removed and s.sql’s
statements become the key’s append (replace → append, joining a
standing append there, if any, and keeping the floor record’s other
delta sections). A run that also writes a floor at the key moves
nothing. The report names the clear step migrate no longer renders
and the way back:
moved ref: zip - s.sql is now a delta at <anchor>, not the whole table (replace → append)
note: rows the table held that s.sql does not are no longer deleted at migrate
to restore the whole table: retrofit data capture --schema app -- s.sql
({"type":"data-moved",…} with table, anchor, labels,
src_file, from, to and path under --as json). Only the floor’s
own source moves it: another source restating the floor’s rows is still
the silent duplicate skip, and a source from stdin has no identity to
move by. The way back holds at identical rows too: data capture s.sql
on the moved key writes the floor again and reports the move back
(overwrote @<anchor> (append → replace), or supersedes once HEAD has
moved; see retrofit data capture).
--dry-run reports the move as would move … and writes nothing.
--purge-superseded opts out of keeping superseded cross-position
blocks, except a witnessed record, sealed or in the tail, which the
purge never touches: each is left byte-intact and reported as
kept: <schema.table>@<anchor> <labels> <path> (witnessed by <opid>; …)
({"type":"data-file-kept",…} under --as json).
Related
- Work with row data: list the rows the emit verbs apply, and write a record by hand.
retrofit import: the flags that classify DML as it is imported.retrofit snapshot: where the band is placed in the DDL, and which records fire triggers.retrofit migrate: how the band is placed within a migration.