retrofit
0.14.0
retrofit.sh

Gate merges on GitHub and GitLab

The steps in Check a pull request in CI gate what reaches main only when they run on the commit the host merges. This page sets them up on each host: the GitHub Actions triggers and their merge queue, a job that tells a conflicted pull request what to do, a job that merges each open pull request with main when main moves, a scheduled job that supplies op compact --frontier, the GitLab pipeline, and a deploy from a Seal.

The examples assume the history bundle script saved in the repository as ci/oplog-bundle.sh, the three-file step saved as ci/oplog-three-file.sh, and one schema, sales, whose oplog is sales.oplog at the repository root, as in Merge branches that change the oplog. Adapt each workflow to the project before you use it. The bundle script passes --allow-database-apply, so a job that runs it also runs the scratch database step. What each path promises lists what each host and git path gives.

Run the checks in GitHub Actions

The script reads three roles from its environment: CANDIDATE, the commit the host tests; TARGET, main’s tip; and BEFORE, the push’s previous tip when there is one. Each GitHub event names them in its own place:

# .github/workflows/oplog.yml
name: oplog
on:
  pull_request:               # each push to a pull request: its merge ref
  merge_group:                # a merge queue's group: the queue's required check
  push:
    branches: [main]
jobs:
  oplog:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0      # the whole history: the bundle refuses a shallow clone
      - name: install the CLI
        run: |
          curl -fsSL https://retrofit.sh/install | sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - name: check the oplog's history
        env:
          EVENT_BEFORE: ${{ github.event.before }}
          QUEUE_BASE: ${{ github.event.merge_group.base_sha }}
        run: |
          CANDIDATE=$GITHUB_SHA
          BEFORE=
          case "$GITHUB_EVENT_NAME" in
            pull_request)
              TARGET=$(git rev-parse "$GITHUB_SHA^1")   # the merge ref's first parent is main
              BEFORE=$EVENT_BEFORE ;;                   # the branch's previous head, on a push to it
            merge_group)
              TARGET=$QUEUE_BASE ;;
            push)
              TARGET=$EVENT_BEFORE ;;                   # main before this push
          esac
          git cat-file -e "$BEFORE^{commit}" 2>/dev/null || BEFORE=   # gone after a force push
          export CANDIDATE TARGET BEFORE
          sh ./ci/oplog-bundle.sh
      - name: merge main's tip and the pull request's head
        if: github.event_name != 'push'
        env:
          PR_SHA: ${{ github.event.pull_request.head.sha }}
          QUEUE_BASE: ${{ github.event.merge_group.base_sha }}
          QUEUE_REF: ${{ github.event.merge_group.head_ref }}
        run: |
          case "$GITHUB_EVENT_NAME" in
            pull_request)
              TARGET=$(git rev-parse "$GITHUB_SHA^1")
              PR_HEAD=$PR_SHA ;;
            merge_group)
              TARGET=$QUEUE_BASE
              n=${QUEUE_REF##*/pr-}; n=${n%%-*}            # refs/heads/gh-readonly-queue/main/pr-<n>-<base>
              git fetch --quiet origin "pull/$n/head"
              PR_HEAD=$(git rev-parse FETCH_HEAD) ;;
          esac
          export TARGET PR_HEAD
          sh ./ci/oplog-three-file.sh
  • fetch-depth: 0. actions/checkout fetches one commit by default, and the history checks need the whole history: from a shallow clone they check nothing and the step fails.
  • merge_group. A merge queue reports a required check only from a workflow that runs on merge_group; without the trigger, the queue waits for a check that never reports and the entry fails. With it, the queue merges the very commit its group tested, under the merge, squash and rebase methods alike.
  • push to main. A queue may merge several entries in one update of main, so a run on main’s push sees only the new tip. The required check on merge_group is the gate on each merged commit; the push run catches what reached main without a group, such as an administrator’s merge.
  • The pull request’s head. The three-file step reads the head its author pushed, not the commit the host tests. On pull_request the event names it. A merge queue’s group commit holds it only under the merge method; under squash and rebase the group commit has one parent. The group’s ref names the pull request’s number, so the step fetches the head by that number. A push to main has no pull request, and the step does not run there; the job in Check open pull requests when main moves runs it.
  • Without a queue, the pull request’s merge ref is what the run tests, and the merge button builds a new commit. Turn on “Require branches to be up to date before merging” so that commit merges the same main the run tested.

Make the job’s check required on main. A team that merges by fast-forward and wants every commit valid sets EVERY_COMMIT=1 on the script’s step (Check every commit of a pull request). Add the CI page’s other steps after these: the data files, the scratch database (a postgres service container), and the annotations under if: failure().

GitHub’s merges read no .gitattributes, so two pull requests that each append to an oplog conflict as text once the first merges. In a merge queue, the second leaves the queue with merge_conflict at the moment the first merges, before any check runs on it, and it stays open. No pull_request run reaches its author; the next section’s job does.

Tell a conflicted pull request what to do

GitHub runs no pull_request workflow on a pull request with a merge conflict, and no pull request event fires when main moves, which is when a pull request usually turns conflicted. pull_request_target does run on a conflicted pull request, in the base repository’s context with a token that can write. So this job runs on main’s push, to check every open pull request, and on pull_request_target, to check one pushed while conflicted. It reads the pull requests through the API as data and never checks out their code: a pull_request_target job that checks out or runs a pull request’s code hands that code its token.

# .github/workflows/oplog-conflict.yml
name: oplog-conflict
on:
  push:
    branches: [main]          # main moved: a pull request may now conflict
  pull_request_target:        # a push to a pull request, conflicted or not
    branches: [main]
permissions:
  contents: read
  pull-requests: write
jobs:
  tell:
    runs-on: ubuntu-latest
    steps:                    # no checkout: nothing from a pull request runs here
      - env:
          GH_TOKEN: ${{ github.token }}
          GH_REPO: ${{ github.repository }}
          PR: ${{ github.event.pull_request.number }}
        run: |
          marker='<!-- oplog-conflict -->'
          cat > comment.md <<EOF
          $marker
          This pull request changes an oplog and conflicts with main. GitHub's
          merges do not read .gitattributes, so two pull requests that append
          to one oplog conflict once the first lands. If the oplog is the file
          that conflicts, resolve it on this branch: bring main into it
          locally, where merge=union keeps both sides' lines, then close the
          fork with retrofit reconcile with main's operations first, and check
          it with retrofit validate before you push. The merging guide shows each
          step: https://docs.retrofit.sh/current/merging/#a-conflicted-pull-request-on-github
          EOF
          prs=${PR:-$(gh pr list --base main --state open --limit 1000 --json number --jq '.[].number')}
          for n in $prs; do
            state=UNKNOWN
            for _ in 1 2 3 4 5 6; do       # GitHub computes mergeable after the event
              state=$(gh pr view "$n" --json mergeable --jq .mergeable)
              [ "$state" != UNKNOWN ] && break
              sleep 10
            done
            [ "$state" = CONFLICTING ] || continue
            gh pr view "$n" --json files --jq '.files[].path' | grep -q '\.oplog$' || continue
            gh pr view "$n" --json comments --jq '.comments[].body' | grep -qF "$marker" && continue
            gh pr comment "$n" --body-file comment.md
          done

GitHub computes mergeable in the background and reports UNKNOWN until it has, so the job asks again before it gives up on a pull request. It comments once per pull request, finding its own comment by the marker. A pull request’s number is the only value it takes from the event.

Check open pull requests when main moves

No pull request event fires when main moves, so a change main takes while a pull request is open meets that pull request’s own change only when the pull request next runs. The three-file step asks for a run on main’s push too, once for each open pull request that changes an oplog, with main’s new tip as TARGET. This job runs it there:

# .github/workflows/oplog-three-file.yml
name: oplog-three-file
on:
  push:
    branches: [main]          # main moved: its change may meet an open pull request's
permissions:
  contents: read
  pull-requests: write
jobs:
  merge:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4   # main's commit: the script and the CLI come from main
        with:
          fetch-depth: 0
      - name: install the CLI
        run: |
          curl -fsSL https://retrofit.sh/install | sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - env:
          GH_TOKEN: ${{ github.token }}
        run: |
          marker='<!-- oplog-three-file -->'
          cat > comment.md <<EOF
          $marker
          This pull request sets a property main also set, and no fork in the
          oplog decides which value stands. Bring main into this branch with
          retrofit reconcile over the three files with --keep and main's oplog
          as --ours, then run the remedy the refusal names after --keep. Where
          --keep already ends the property with main's value, skip the remedy.
          Where the refusal names this branch's value after --keep and no
          remedy, --keep has still written that value over main's: revise this
          branch's operation to main's value, or remove it where that leaves
          main's value standing, and run retrofit validate. If neither
          validates, do not commit the merge.
          After a rebase, run the three files too and commit the result on top
          of the replayed commits. Check the result with retrofit validate
          before you push. The refusal is in this run's log: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID
          The CI guide shows each step: https://docs.retrofit.sh/current/ci/#mains-tip-against-the-pull-requests-head
          EOF
          gh pr list --base main --state open --limit 1000 --json number,headRefOid \
            --jq '.[] | "\(.number) \(.headRefOid)"' > "$RUNNER_TEMP/prs"
          git fetch --quiet origin '+refs/pull/*/head:refs/remotes/origin/pull/*'   # heads from forks
          failed=0
          while read -r n head <&3; do
            gh pr view "$n" --json files --jq '.files[].path' | grep -q '\.oplog$' || continue
            git cat-file -e "$head^{commit}" 2>/dev/null || continue   # pushed again: that push's run checks it
            rc=0
            TARGET=$GITHUB_SHA PR_HEAD=$head sh ./ci/oplog-three-file.sh || rc=$?
            case $rc in
              0) continue ;;
              1) ;;                                     # a refusal: tell the pull request
              *) failed=1; continue ;;                  # not a refusal: the job fails
            esac
            gh pr view "$n" --json comments --jq '.comments[].body' | grep -qF "$marker" && continue
            gh pr comment "$n" --body-file comment.md
          done 3< "$RUNNER_TEMP/prs"
          exit "$failed"
  • Main’s code only. The job runs on main’s push and never on pull_request_target. It checks out main and runs main’s script; a pull request’s oplogs reach it as files the script reads with git show, and nothing from a pull request runs. A pull request from a fork (a copy of the repository under another account) has its head only under refs/pull/<n>/head, which the fetch brings in.
  • What it reports. When the merge stops on a conflict, the job comments once on the pull request, finding its own comment by the marker, and passes: main is not what failed. Any other failure fails the job. The list comes first, so a head a later push moved on still arrives; a head the fetch lacks, after a force push, is skipped: that push’s own pull_request run checks it.

Compact the oplog to the open pull requests’ frontier

op compact --frontier <file> drops a trace only when <file> holds the same trace, so every open branch already holds what is dropped. The file is main’s oplog at the stable frontier: the oldest point where an open pull request forked from main, drafts and pull requests from forks included, which every open pull request holds. A scheduled job finds the frontier and opens a pull request with the result:

# .github/workflows/oplog-frontier.yml
name: oplog-frontier
on:
  schedule:
    - cron: "0 6 * * 1"
  workflow_dispatch:
jobs:
  compact:
    runs-on: ubuntu-latest
    permissions:
      contents: read          # the bot's token pushes and opens the pull request
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          token: ${{ secrets.OPLOG_BOT_TOKEN }}     # git push uses the token checkout persists
      - name: install the CLI
        run: |
          curl -fsSL https://retrofit.sh/install | sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - env:
          GH_TOKEN: ${{ secrets.OPLOG_BOT_TOKEN }}   # a GitHub App's or a bot's token, not github.token
        run: |
          set -eu
          gh pr list --state open --limit 1000 --json headRefOid --jq '.[].headRefOid' > "$RUNNER_TEMP/heads"
          [ "$(wc -l < "$RUNNER_TEMP/heads")" -lt 1000 ] || { echo "1000 or more open pull requests: the list may be cut" >&2; exit 1; }
          git fetch --quiet origin '+refs/pull/*/head:refs/remotes/origin/pull/*'   # heads from forks
          : > "$RUNNER_TEMP/fork-points"
          while read -r head; do                          # a head not fetched stops the job: no partial frontier
            git merge-base origin/main "$head" >> "$RUNNER_TEMP/fork-points"
          done < "$RUNNER_TEMP/heads"
          [ -s "$RUNNER_TEMP/fork-points" ] || exit 0   # no open pull request
          frontier=$(git merge-base --octopus $(sort -u "$RUNNER_TEMP/fork-points"))   # in every pull request's history
          git show "$frontier:sales.oplog" > "$RUNNER_TEMP/frontier.oplog"
          retrofit op compact --frontier "$RUNNER_TEMP/frontier.oplog"
          git diff --quiet -- sales.oplog && exit 0      # nothing to drop: the file is unchanged
          git switch -c oplog-compact
          git -c user.name=oplog-bot -c user.email=oplog-bot@users.noreply.github.com \
            commit -q -m "Compact the oplog to the open pull requests' frontier" -- sales.oplog
          git push -f origin oplog-compact
          gh pr view oplog-compact --json state --jq .state 2>/dev/null | grep -qx OPEN ||
            gh pr create --base main --head oplog-compact \
              --title "Compact the oplog" --body "Frontier: $frontier"
  • gh pr list lists drafts with the rest; --limit lifts its default of 30. A pull request from a fork has its head only under refs/pull/<n>/head, which the fetch brings in so git merge-base can read it. The list comes first, so a listed head that a later push moved on still arrives, as an ancestor of the new head. A listed head the fetch lacks, after a force push, stops the job rather than compacting to a frontier that leaves its pull request out.
  • The frontier is the newest commit every fork point descends from, so every open pull request holds it. On a main without merge commits that is the oldest fork point; where main merged parallel histories, the oldest fork point in one may not be in a pull request forked from the other.
  • A list that reaches --limit may be incomplete, and a pull request left out of it may hold a trace the job would drop, so the job stops there instead.
  • A branch pushed with github.token starts no workflow run, and a pull request it opens starts none on its own, at most one that waits for approval, so the result can go unchecked. Give a GitHub App’s or a bot user’s token to both actions/checkout, whose token git push uses, and gh.
  • In a project with more than one schema, run op compact --schema <name> once per schema, with that schema’s oplog at the frontier. Name a Seal with --seal to compact its sealed range too.

A branch the job cannot see, such as one never pushed, may still hold a dropped trace. The history checks fail its pull request when it opens.

Run the checks in GitLab CI

GitLab’s merges honour merge=union: two merge requests that each append to an oplog merge without a conflict and leave a fork. Nobody reconciles main. validate fails the fork, so the history check in the pipeline that tests the merge result keeps it out of main, and the fork is closed on the merge request’s branch.

# .gitlab-ci.yml
oplog:
  variables:
    GIT_DEPTH: 0              # the whole history: the default of 20 commits refuses as shallow
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"   # the merge request's pipeline, a train's car
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH   # branch pushes: the only pipelines with a before
  before_script:
    - curl -fsSL https://retrofit.sh/install | sh
    - export PATH="$HOME/.local/bin:$PATH"
  script:
    - git fetch --quiet origin "$CI_DEFAULT_BRANCH"
    - |
      CANDIDATE=$CI_COMMIT_SHA
      BEFORE=
      if [ -n "${CI_MERGE_REQUEST_IID:-}" ]; then
        TARGET=${CI_MERGE_REQUEST_TARGET_BRANCH_SHA:-$(git rev-parse FETCH_HEAD)}
      elif [ "$CI_COMMIT_BRANCH" = "$CI_DEFAULT_BRANCH" ]; then
        TARGET=$CI_COMMIT_BEFORE_SHA                     # main before this push
      else
        TARGET=$(git rev-parse FETCH_HEAD)
        BEFORE=$CI_COMMIT_BEFORE_SHA                     # all zeros on a branch's first push
      fi
      git cat-file -e "$BEFORE^{commit}" 2>/dev/null || BEFORE=
      export CANDIDATE TARGET BEFORE
      sh ./ci/oplog-bundle.sh
      if [ -n "${CI_MERGE_REQUEST_IID:-}" ]; then   # the three-file step: merge request pipelines only
        PR_HEAD=${CI_MERGE_REQUEST_SOURCE_BRANCH_SHA:-$CI_COMMIT_SHA}
        export PR_HEAD
        sh ./ci/oplog-three-file.sh
      fi

The job runs on an image with git, curl and jq.

  • GIT_DEPTH: 0. A new GitLab project clones 20 commits by default, and the history checks need the whole history.
  • Branch pipelines. CI_COMMIT_BEFORE_SHA is all zeros in every merge request pipeline, and only a branch pipeline started by a push has the push’s previous tip, which one compact-form check reads. So the job also runs on push pipelines; a workflow:rules that drops duplicate pipelines must keep them. A scheduled or manual pipeline has no previous tip either, so the rule leaves it out.
  • What gates the merge. “Pipelines must succeed” checks only the merge request pipeline, so a branch pipeline’s failure of that check shows on the commit and does not block the merge. In the merge request pipeline, that check is reported unchecked.
  • The target. CI_MERGE_REQUEST_TARGET_BRANCH_SHA is set only in a merged results pipeline, a train’s car included; elsewhere the job fetches main for its tip.
  • The merge request’s head. CI_MERGE_REQUEST_SOURCE_BRANCH_SHA is likewise set only in a merged results pipeline, whose commit is the temporary merge. In any other merge request pipeline, the commit the pipeline tests is the head itself. Main’s moving starts no merge request pipeline, so GitLab has no counterpart to GitHub’s job on main’s push: the merge request’s pipeline and a train’s car run the three-file step before the merge request merges.

For the annotations, add the code quality conversion in after_script and name its file under artifacts:reports:codequality, with artifacts:when: always so a failed job still reports it.

Use a merge train, or semi-linear or fast-forward merges with “Pipelines must succeed” and automatic rebase off: the check gates what merges only where the pipeline tests the commit that merges. GitLab setups lists what each setup tests and what becomes of a fork on it.

A train cannot fix a car: a push to its branch drops it from the train. Close the fork on the branch before the train, with main in the branch’s history, so the next merge is trivial: bring main in and write the oplog from the three files, or close the fork in the one file where they are not at hand. After a rebase, use them too, and commit the result on top of the replayed commits. Two concurrent oplog merge requests in one train leave the second forked; it leaves the train, and every car behind it restarts. Add one oplog merge request to the train at a time to avoid the restarts.

On a setup that does not test what merges, more merge requests can merge before main is fixed, and the fork grows a block with each. reconcile closes a fork of three or more blocks two at a time, in line order, on a branch; the result goes through the pipeline like any other change.

Deploy and catch up a database

Deploy from a Seal: an unsealed operation’s position moves as the log grows (Sealed and unsealed ops). When a merge’s order is forced, the pull request’s operations are written before main’s, and a database deployed from main’s unsealed tail before that merge is left behind.

A database that takes only sealed operations catches up to the latest Seal from a dump of itself:

pg_dump --schema-only "$DATABASE_URL" > db.sql
retrofit migrate --from-dump db.sql --to seal > deploy.sql
psql -X -v ON_ERROR_STOP=1 -f deploy.sql "$DATABASE_URL"

A database deployed from the unsealed tail is a working copy. Check it with identify, and where it reports the database behind, rebuild it from the log, its rows saved first: render the whole history into a fresh database, then move the rows across.

pg_dump --schema-only "$DATABASE_URL" > db.sql
retrofit identify db.sql          # reports it behind, naming the operations after its nearest cut
retrofit migrate --from root --to last > rebuild.sql
psql -X -v ON_ERROR_STOP=1 -f rebuild.sql "$FRESH_DATABASE_URL"

Neither identify nor migrate --from-dump offers a catch-up for such a database: the operations a forced merge put first do not replay over a database that already holds main’s.

Before 1.0, any release may change behavior; the release notes list what changed.

Published .