retrofit
0.13.0
retrofit.sh

Sealing and deploying

How a range of the oplog becomes stable history, how you track where each database is, and what to do when one changes outside the oplog.

Sealed and unsealed ops

The oplog has two regions: sealed and unsealed.

  • Unsealed tail: the working surface. You can append (op add), edit (op revise), or remove (op remove) ops freely. You can open the editor buffer (op revise -i) and delete lines. No confirmation step guards it, because no database has applied it. That holds only while nobody has, so a tail shared through git is sealed before anyone applies it.
  • Sealed range: frozen history. Once you op seal, the range is locked: a Merkle digest over the sealed ops’ canonical bytes is recorded, and any in-place edit to a sealed op breaks verification. A seal is the team’s agreement that the range is stable, because branches and databases build on it; it records no deployment.

Seal a range before anyone else applies it. New work goes on a fresh unsealed tail behind the seal. migrate starts from the latest seal by default, because the agreement makes it the expected baseline; a database behind it names its cut with --from.

Where each database is

Which cut each of your databases is at is yours to record: retrofit emits one log for many databases and connects to none of them, so the log holds no database’s position. A database promoted on its own schedule, such as production behind a UAT database that takes every seal, can keep its position in a committed file per environment (deploy/prod.cut) holding the opid it was last migrated to. That opid is any cut of the log, not necessarily a seal. Before a deploy, dump the schema (pg_dump --schema-only) and check it against the file with identify, which reports the cut of the log a dump is at. To promote, emit the range from the recorded cut and write the new opid back once it is applied:

set -eo pipefail   # bash: any failure stops the script before the write-back
retrofit identify prod.sql --expect "$(cat deploy/prod.cut)"
retrofit migrate --from "$(cat deploy/prod.cut)" --to 7a6b1234 \
  | psql -X -v ON_ERROR_STOP=1 "$PROD_URL"
echo 7a6b1234 > deploy/prod.cut

Each step runs only if the one before it succeeded: a mismatched dump stops the migrate, and a failed statement stops the write-back, so the file never records a cut the database has not reached.

With --expect, identify passes only when the dump is at the cut you name. A dump anywhere else, including one with a change made by hand, fails and reports where the dump is instead. The match is of schema: it says the dump is at that cut, not that the database applied those ops. identify has the rest.

The file and its write-back are yours; retrofit reads the opid you pass and writes neither. Recovery is git: put the file back from version control.

When a database changes outside the oplog

A database drifts when someone changes its schema without the oplog: a hotfix applied by hand, an index added from a console. The oplog no longer describes that database, so the next migration is written for a schema it does not have, and fails or does the wrong thing. identify finds drift before a deploy does; drift records it in the oplog.

What identify reports

identify compares a schema dump (pg_dump --schema-only) with every cut of the oplog and gives one of these answers:

  • At a cut. The dump matches one point in the oplog, and identify prints its opid, or its Seal. Nothing has drifted.
  • Changed at the head. The dump matches no cut, and the nearest is the last op: the database was current, then someone changed it. identify lists the difference and names drift --dry-run to record it.
  • Behind and changed. The nearest cut has ops after it that the dump lacks, and the dump differs from that cut as well. drift compares with the last op, so it would record the reversal of those ops, and identify does not offer it. The database is rebuilt from the log, its rows saved first.
  • Changed by an unsealed op. The difference is in an element an unsealed op created or changed, as when the database applied an op that was later revised. As with a database behind, the database is rebuilt from the log.
  • More than one cut. The dump matches several points, such as one before an op and one after the op that undoes it. identify lists them, and --expect confirms the one you recorded.

Every answer but the first exits non-zero, so identify run on a schedule, or before each deploy, is a drift check.

What a schema dump cannot show

identify and drift see what a schema dump holds and retrofit models: tables, columns, constraints, indexes, views, functions and the other elements Operators lists. Drift outside that is unseen:

  • Rows. A schema dump holds none, so a changed row in a reference table, or a backfill that never ran, does not show.
  • RunRawScript effects. A DO block changes no schema element, so identify notes how many lie before the matched cut and that their effects cannot be confirmed.
  • Objects retrofit does not model, such as privileges and sequences no column owns. identify notes how many took no part in the match.
  • Objects the project excludes (Exclude objects the project does not own).
  • An indexed partition tree. Neither identify nor drift compares a partition tree whose partitioned table holds an index: identify cannot place a database holding one, drift refuses the run, and op add warns schema-partition-tree-indexed when an op leaves an index on a partitioned table. Where the tree is the only difference, identify says so; name the cut the database is at by hand, with migrate --from (--at under --down) in place of --from-dump.
  • How the database got there. A match says the schema is at that cut, not that the database applied those ops.

Record the drift

When identify names drift, preview what it would record, then record it: drift appends the ops to the unsealed tail, where you review them and seal them, and the database is at the new last op. Record a schema changed outside the oplog has the steps.

Before 1.0, any release may change behavior; the release notes list what changed.

Published .