Recover after a failed apply or a merge
Two things can leave work half done: a migration script that stopped part way through, and a merge that left two histories side by side in the oplog. This page gets you from each back to a state every command accepts.
Find out what a failed apply left
What a failure leaves depends on whether the script applied as one transaction:
- A script that is one transaction (the default, unless it commits part
way; or a
callerfile applied inside your own transaction) leaves nothing applied. Fix the cause and apply the same file again. - A script that commits part way leaves every part before the failure
committed. When you emitted it, stderr said it is not all-or-nothing;
in the file, each part ends with a
COMMIT;and the next opens withBEGIN;. Applying the same file again fails at the first statement that already applied. - A file emitted with
--envelope=independentcommits each statement on its own, so every statement before the failure has applied.
psql names the line it stopped at (psql:migration.sql:<line>: ERROR: …). Every COMMIT; above that line has taken effect.
Finish a migration that stopped part way
Fix what made the statement fail, then emit the rest of the migration from where the database now is, and apply that:
pg_dump --schema-only -d <database> > db.sql
retrofit migrate --from-dump db.sql > rest.sql
psql -X -v ON_ERROR_STOP=1 -d <database> -f rest.sql
The dump places the database after the last part that committed, so
rest.sql holds only what did not apply. Pass the same --to,
envelope and --label flags the first script had. When you know the last op
whose part committed, --from <opid> names it directly in place of
--from-dump. Starting from a
dump covers a dump that
matches no point in the log.
A reverse script that stopped part way finishes the same way in its own
direction: keep --down and the same --from, the point you are
rolling back to, and the dump fills --at:
retrofit migrate --down --from <target> --from-dump db.sql > rest.sql
Without --down, the rest would run forward and put back what the
reverse script already removed.
A schema-only dump does not show rows. rest.sql starts after the
last schema change the dump shows, so data rows that follow that change
in the original file and did not apply can be missing from it. Compare
the two files before you apply rest.sql, and apply those rows from the
original file by hand, in the place each held there:
- Rows that come before a statement
rest.sqlalso holds go first, beforerest.sql. - Rows in the seed block, which follows every schema change, go last,
after
rest.sql: their statements name tables and columns as the last schema change leaves them.
When the part that failed held only data rows, rest.sql is empty and
those rows are all that is left. Which rows did not apply depends on the
file:
- In a file that commits part way, the failed part rolled back, so no
row from its
BEGIN;on applied. Open the rows you apply withBEGIN;and close them withCOMMIT;. - In an
independentfile, each statement before the failed one has committed, and the file has noBEGIN;orCOMMIT;. Start at the statement that failed; applying an earlierINSERTagain repeats its rows. When that statement names apg_temp."retrofit_replace_…"table, it is part of a staged block, and the temp table ended with the failed session: start at the block’sDROP TABLE IF EXISTSline instead. The block replaces the table’s rows as a whole, so running it again is safe (The replace prologue).
Reconcile after a merge
When two branches each appended ops and git merged them, the oplog holds
both branches’ new ops after their shared op: a
fork. A command that reads the oplog then prints an
error and a hint: line naming the next step. To close the fork:
retrofit validate # reports the fork the merge left
retrofit reconcile # reports the order it would write; writes nothing
retrofit reconcile --clean --swap # writes main's ops first when yours come first in the file
retrofit validate # passes
Write main’s ops first: they are the ones databases have already
applied. After a rebase onto main, main’s lines already come first in
the file, so retrofit reconcile --clean --keep writes that order where it
reports overrides (a bare --clean refuses fork-forced-order). Where you
have the three versions of the file (the merge base, main’s and your
branch’s), write the result from them instead, with the three-file
merge; Resolve
conflict markers in the oplog
covers an oplog that holds conflict markers.
When neither order replays cleanly, reconcile --clean writes nothing
and names the rule the order breaks. Write the order anyway with
retrofit reconcile --accept-dirty, adding --swap as before when your
ops come first in the file, then fix the ops that break it with
retrofit op revise, and run validate again.
A Seal in the fork decides more of the order: when one block holds a
Seal, that block is written first and reconcile refuses --keep or
--swap, so run retrofit reconcile --clean without either, adding the flag that
names the Seal’s order when the refusal fork-forced-order asks for it. Two
sealed blocks need you to say which release stands.
Ship a hotfix ahead of the next release walks that case,
and Refusals after a merge maps
each error a fork prints to the command that clears it.
Related
- Emit a migration script.
- Choose who opens the transaction: a script that applies as one transaction leaves nothing to recover.
- Bound how long a migration waits for a lock.
- Merge branches that change the oplog: conflict markers, the three-file merge and jj.
- How a migration applies: why some scripts commit in parts.