retrofit
0.14.0
retrofit.sh

Recover after a failed apply or a merge

Two things can leave work half done: a migration script that stopped part way through, and a merge that left two histories side by side in the oplog. This page gets you from each back to a state every command accepts.

Find out what a failed apply left

What a failure leaves depends on whether the script applied as one transaction:

  • A script that is one transaction (the default, unless it commits part way; or a caller file applied inside your own transaction) leaves nothing applied. Fix the cause and apply the same file again.
  • A script that commits part way leaves every part before the failure committed. When you emitted it, stderr said it is not all-or-nothing; in the file, each part ends with a COMMIT; and the next opens with BEGIN;. Applying the same file again fails at the first statement that already applied.
  • A file emitted with --envelope=independent commits each statement on its own, so every statement before the failure has applied.

psql names the line it stopped at (psql:migration.sql:<line>: ERROR: …). Every COMMIT; above that line has taken effect.

Finish a migration that stopped part way

Fix what made the statement fail, then emit the rest of the migration from where the database now is, and apply that:

pg_dump --schema-only -d <database> > db.sql
retrofit migrate --from-dump db.sql > rest.sql
psql -X -v ON_ERROR_STOP=1 -d <database> -f rest.sql

The dump places the database after the last part that committed, so rest.sql holds only what did not apply. Pass the same --to, envelope and --label flags the first script had. When you know the last op whose part committed, --from <opid> names it directly in place of --from-dump. Starting from a dump covers a dump that matches no point in the log.

A reverse script that stopped part way finishes the same way in its own direction: keep --down and the same --from, the point you are rolling back to, and the dump fills --at:

retrofit migrate --down --from <target> --from-dump db.sql > rest.sql

Without --down, the rest would run forward and put back what the reverse script already removed.

A schema-only dump does not show rows. rest.sql starts after the last schema change the dump shows, so data rows that follow that change in the original file and did not apply can be missing from it. Compare the two files before you apply rest.sql, and apply those rows from the original file by hand, in the place each held there:

  • Rows that come before a statement rest.sql also holds go first, before rest.sql.
  • Rows in the seed block, which follows every schema change, go last, after rest.sql: their statements name tables and columns as the last schema change leaves them.

When the part that failed held only data rows, rest.sql is empty and those rows are all that is left. Which rows did not apply depends on the file:

  • In a file that commits part way, the failed part rolled back, so no row from its BEGIN; on applied. Open the rows you apply with BEGIN; and close them with COMMIT;.
  • In an independent file, each statement before the failed one has committed, and the file has no BEGIN; or COMMIT;. Start at the statement that failed; applying an earlier INSERT again repeats its rows. When that statement names a pg_temp."retrofit_replace_…" table, it is part of a staged block, and the temp table ended with the failed session: start at the block’s DROP TABLE IF EXISTS line instead. The block replaces the table’s rows as a whole, so running it again is safe (The replace prologue).

Reconcile after a merge

When two branches each appended ops and git merged them, the oplog holds both branches’ new ops after their shared op: a fork. A command that reads the oplog then prints an error and a hint: line naming the next step. To close the fork:

retrofit validate                    # reports the fork the merge left
retrofit reconcile                   # reports the order it would write; writes nothing
retrofit reconcile --clean --swap    # writes main's ops first when yours come first in the file
retrofit validate                    # passes

Write main’s ops first: they are the ones databases have already applied. After a rebase onto main, main’s lines already come first in the file, so retrofit reconcile --clean --keep writes that order where it reports overrides (a bare --clean refuses fork-forced-order). Where you have the three versions of the file (the merge base, main’s and your branch’s), write the result from them instead, with the three-file merge; Resolve conflict markers in the oplog covers an oplog that holds conflict markers.

When neither order replays cleanly, reconcile --clean writes nothing and names the rule the order breaks. Write the order anyway with retrofit reconcile --accept-dirty, adding --swap as before when your ops come first in the file, then fix the ops that break it with retrofit op revise, and run validate again.

A Seal in the fork decides more of the order: when one block holds a Seal, that block is written first and reconcile refuses --keep or --swap, so run retrofit reconcile --clean without either, adding the flag that names the Seal’s order when the refusal fork-forced-order asks for it. Two sealed blocks need you to say which release stands. Ship a hotfix ahead of the next release walks that case, and Refusals after a merge maps each error a fork prints to the command that clears it.

Before 1.0, any release may change behavior; the release notes list what changed.

Published .