Gate merges on GitHub and GitLab
The steps in Check a pull request in CI gate what reaches main
only when they run on the commit the host merges. This page sets them up
on each host: the GitHub Actions triggers and their merge queue, a job
that tells a conflicted pull request what to do, a job that merges each
open pull request with main when main moves, a scheduled job that
supplies op compact --frontier, the GitLab pipeline, and a deploy from
a Seal.
The examples assume
the history bundle script
saved in the repository as ci/oplog-bundle.sh,
the three-file step
saved as ci/oplog-three-file.sh, and one schema, sales, whose
oplog is sales.oplog at the repository root, as
in Merge branches that change the oplog. Adapt each
workflow to the project before you use it. The bundle script passes
--allow-database-apply, so a job that runs it also runs
the scratch database step.
What each path promises
lists what each host and git path gives.
Run the checks in GitHub Actions
The script reads three roles from its environment: CANDIDATE, the
commit the host tests; TARGET, main’s tip; and BEFORE, the push’s
previous tip when there is one. Each GitHub event names them in its own
place:
# .github/workflows/oplog.yml
name: oplog
on:
pull_request: # each push to a pull request: its merge ref
merge_group: # a merge queue's group: the queue's required check
push:
branches: [main]
jobs:
oplog:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # the whole history: the bundle refuses a shallow clone
- name: install the CLI
run: |
curl -fsSL https://retrofit.sh/install | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: check the oplog's history
env:
EVENT_BEFORE: ${{ github.event.before }}
QUEUE_BASE: ${{ github.event.merge_group.base_sha }}
run: |
CANDIDATE=$GITHUB_SHA
BEFORE=
case "$GITHUB_EVENT_NAME" in
pull_request)
TARGET=$(git rev-parse "$GITHUB_SHA^1") # the merge ref's first parent is main
BEFORE=$EVENT_BEFORE ;; # the branch's previous head, on a push to it
merge_group)
TARGET=$QUEUE_BASE ;;
push)
TARGET=$EVENT_BEFORE ;; # main before this push
esac
git cat-file -e "$BEFORE^{commit}" 2>/dev/null || BEFORE= # gone after a force push
export CANDIDATE TARGET BEFORE
sh ./ci/oplog-bundle.sh
- name: merge main's tip and the pull request's head
if: github.event_name != 'push'
env:
PR_SHA: ${{ github.event.pull_request.head.sha }}
QUEUE_BASE: ${{ github.event.merge_group.base_sha }}
QUEUE_REF: ${{ github.event.merge_group.head_ref }}
run: |
case "$GITHUB_EVENT_NAME" in
pull_request)
TARGET=$(git rev-parse "$GITHUB_SHA^1")
PR_HEAD=$PR_SHA ;;
merge_group)
TARGET=$QUEUE_BASE
n=${QUEUE_REF##*/pr-}; n=${n%%-*} # refs/heads/gh-readonly-queue/main/pr-<n>-<base>
git fetch --quiet origin "pull/$n/head"
PR_HEAD=$(git rev-parse FETCH_HEAD) ;;
esac
export TARGET PR_HEAD
sh ./ci/oplog-three-file.sh
fetch-depth: 0.actions/checkoutfetches one commit by default, and the history checks need the whole history: from a shallow clone they check nothing and the step fails.merge_group. A merge queue reports a required check only from a workflow that runs onmerge_group; without the trigger, the queue waits for a check that never reports and the entry fails. With it, the queue merges the very commit its group tested, under the merge, squash and rebase methods alike.pushto main. A queue may merge several entries in one update of main, so a run on main’s push sees only the new tip. The required check onmerge_groupis the gate on each merged commit; the push run catches what reached main without a group, such as an administrator’s merge.- The pull request’s head. The three-file
step reads the head
its author pushed, not the commit the host tests. On
pull_requestthe event names it. A merge queue’s group commit holds it only under the merge method; under squash and rebase the group commit has one parent. The group’s ref names the pull request’s number, so the step fetches the head by that number. A push to main has no pull request, and the step does not run there; the job in Check open pull requests when main moves runs it. - Without a queue, the pull request’s merge ref is what the run tests, and the merge button builds a new commit. Turn on “Require branches to be up to date before merging” so that commit merges the same main the run tested.
Make the job’s check required on main. A team that merges by
fast-forward and wants every commit valid sets EVERY_COMMIT=1 on the
script’s step
(Check every commit of a pull request).
Add the CI page’s other steps after these:
the data files,
the scratch database
(a postgres service container), and
the annotations under
if: failure().
GitHub’s merges read no .gitattributes, so two pull requests that
each append to an oplog conflict as text once the first merges. In a
merge queue, the second leaves the queue with merge_conflict at the
moment the first merges, before any check runs on it, and it stays
open. No pull_request run reaches its author; the next section’s job
does.
Tell a conflicted pull request what to do
GitHub runs no pull_request workflow on a pull request with a merge
conflict, and no pull request event fires when main moves, which is
when a pull request usually turns conflicted. pull_request_target
does run on a conflicted pull request, in the base repository’s context
with a token that can write. So this job runs on main’s push, to check
every open pull request, and on pull_request_target, to check one
pushed while conflicted. It reads the pull requests through the API as
data and never checks out their code: a pull_request_target job that
checks out or runs a pull request’s code hands that code its token.
# .github/workflows/oplog-conflict.yml
name: oplog-conflict
on:
push:
branches: [main] # main moved: a pull request may now conflict
pull_request_target: # a push to a pull request, conflicted or not
branches: [main]
permissions:
contents: read
pull-requests: write
jobs:
tell:
runs-on: ubuntu-latest
steps: # no checkout: nothing from a pull request runs here
- env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
PR: ${{ github.event.pull_request.number }}
run: |
marker='<!-- oplog-conflict -->'
cat > comment.md <<EOF
$marker
This pull request changes an oplog and conflicts with main. GitHub's
merges do not read .gitattributes, so two pull requests that append
to one oplog conflict once the first lands. If the oplog is the file
that conflicts, resolve it on this branch: bring main into it
locally, where merge=union keeps both sides' lines, then close the
fork with retrofit reconcile with main's operations first, and check
it with retrofit validate before you push. The merging guide shows each
step: https://docs.retrofit.sh/current/merging/#a-conflicted-pull-request-on-github
EOF
prs=${PR:-$(gh pr list --base main --state open --limit 1000 --json number --jq '.[].number')}
for n in $prs; do
state=UNKNOWN
for _ in 1 2 3 4 5 6; do # GitHub computes mergeable after the event
state=$(gh pr view "$n" --json mergeable --jq .mergeable)
[ "$state" != UNKNOWN ] && break
sleep 10
done
[ "$state" = CONFLICTING ] || continue
gh pr view "$n" --json files --jq '.files[].path' | grep -q '\.oplog$' || continue
gh pr view "$n" --json comments --jq '.comments[].body' | grep -qF "$marker" && continue
gh pr comment "$n" --body-file comment.md
done
GitHub computes mergeable in the background and reports UNKNOWN
until it has, so the job asks again before it gives up on a pull
request. It comments once per pull request, finding its own comment by
the marker. A pull request’s number is the only value it takes from
the event.
Check open pull requests when main moves
No pull request event fires when main moves, so a change main takes
while a pull request is open meets that pull request’s own change only
when the pull request next runs. The three-file
step asks for a run on
main’s push too, once for each open pull request that changes an oplog,
with main’s new tip as TARGET. This job runs it there:
# .github/workflows/oplog-three-file.yml
name: oplog-three-file
on:
push:
branches: [main] # main moved: its change may meet an open pull request's
permissions:
contents: read
pull-requests: write
jobs:
merge:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4 # main's commit: the script and the CLI come from main
with:
fetch-depth: 0
- name: install the CLI
run: |
curl -fsSL https://retrofit.sh/install | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- env:
GH_TOKEN: ${{ github.token }}
run: |
marker='<!-- oplog-three-file -->'
cat > comment.md <<EOF
$marker
This pull request sets a property main also set, and no fork in the
oplog decides which value stands. Bring main into this branch with
retrofit reconcile over the three files with --keep and main's oplog
as --ours, then run the remedy the refusal names after --keep. Where
--keep already ends the property with main's value, skip the remedy.
Where the refusal names this branch's value after --keep and no
remedy, --keep has still written that value over main's: revise this
branch's operation to main's value, or remove it where that leaves
main's value standing, and run retrofit validate. If neither
validates, do not commit the merge.
After a rebase, run the three files too and commit the result on top
of the replayed commits. Check the result with retrofit validate
before you push. The refusal is in this run's log: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID
The CI guide shows each step: https://docs.retrofit.sh/current/ci/#mains-tip-against-the-pull-requests-head
EOF
gh pr list --base main --state open --limit 1000 --json number,headRefOid \
--jq '.[] | "\(.number) \(.headRefOid)"' > "$RUNNER_TEMP/prs"
git fetch --quiet origin '+refs/pull/*/head:refs/remotes/origin/pull/*' # heads from forks
failed=0
while read -r n head <&3; do
gh pr view "$n" --json files --jq '.files[].path' | grep -q '\.oplog$' || continue
git cat-file -e "$head^{commit}" 2>/dev/null || continue # pushed again: that push's run checks it
rc=0
TARGET=$GITHUB_SHA PR_HEAD=$head sh ./ci/oplog-three-file.sh || rc=$?
case $rc in
0) continue ;;
1) ;; # a refusal: tell the pull request
*) failed=1; continue ;; # not a refusal: the job fails
esac
gh pr view "$n" --json comments --jq '.comments[].body' | grep -qF "$marker" && continue
gh pr comment "$n" --body-file comment.md
done 3< "$RUNNER_TEMP/prs"
exit "$failed"
- Main’s code only. The job runs on main’s push and never on
pull_request_target. It checks out main and runs main’s script; a pull request’s oplogs reach it as files the script reads withgit show, and nothing from a pull request runs. A pull request from a fork (a copy of the repository under another account) has its head only underrefs/pull/<n>/head, which the fetch brings in. - What it reports. When the merge stops on a conflict, the job
comments once on the pull request, finding its own comment by the
marker, and passes: main is not what failed. Any other failure fails
the job. The list comes first, so a head a later push moved on still
arrives; a head the fetch lacks, after a force push, is skipped: that
push’s own
pull_requestrun checks it.
Compact the oplog to the open pull requests’ frontier
op compact --frontier <file> drops a trace only when <file> holds
the same trace, so every open branch already holds what is dropped. The
file is main’s oplog at the stable frontier: the oldest point where an
open pull request forked from main, drafts and pull requests from forks
included, which every open pull request holds. A scheduled job finds
the frontier and opens a pull request with the result:
# .github/workflows/oplog-frontier.yml
name: oplog-frontier
on:
schedule:
- cron: "0 6 * * 1"
workflow_dispatch:
jobs:
compact:
runs-on: ubuntu-latest
permissions:
contents: read # the bot's token pushes and opens the pull request
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.OPLOG_BOT_TOKEN }} # git push uses the token checkout persists
- name: install the CLI
run: |
curl -fsSL https://retrofit.sh/install | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- env:
GH_TOKEN: ${{ secrets.OPLOG_BOT_TOKEN }} # a GitHub App's or a bot's token, not github.token
run: |
set -eu
gh pr list --state open --limit 1000 --json headRefOid --jq '.[].headRefOid' > "$RUNNER_TEMP/heads"
[ "$(wc -l < "$RUNNER_TEMP/heads")" -lt 1000 ] || { echo "1000 or more open pull requests: the list may be cut" >&2; exit 1; }
git fetch --quiet origin '+refs/pull/*/head:refs/remotes/origin/pull/*' # heads from forks
: > "$RUNNER_TEMP/fork-points"
while read -r head; do # a head not fetched stops the job: no partial frontier
git merge-base origin/main "$head" >> "$RUNNER_TEMP/fork-points"
done < "$RUNNER_TEMP/heads"
[ -s "$RUNNER_TEMP/fork-points" ] || exit 0 # no open pull request
frontier=$(git merge-base --octopus $(sort -u "$RUNNER_TEMP/fork-points")) # in every pull request's history
git show "$frontier:sales.oplog" > "$RUNNER_TEMP/frontier.oplog"
retrofit op compact --frontier "$RUNNER_TEMP/frontier.oplog"
git diff --quiet -- sales.oplog && exit 0 # nothing to drop: the file is unchanged
git switch -c oplog-compact
git -c user.name=oplog-bot -c user.email=oplog-bot@users.noreply.github.com \
commit -q -m "Compact the oplog to the open pull requests' frontier" -- sales.oplog
git push -f origin oplog-compact
gh pr view oplog-compact --json state --jq .state 2>/dev/null | grep -qx OPEN ||
gh pr create --base main --head oplog-compact \
--title "Compact the oplog" --body "Frontier: $frontier"
gh pr listlists drafts with the rest;--limitlifts its default of 30. A pull request from a fork has its head only underrefs/pull/<n>/head, which the fetch brings in sogit merge-basecan read it. The list comes first, so a listed head that a later push moved on still arrives, as an ancestor of the new head. A listed head the fetch lacks, after a force push, stops the job rather than compacting to a frontier that leaves its pull request out.- The frontier is the newest commit every fork point descends from, so every open pull request holds it. On a main without merge commits that is the oldest fork point; where main merged parallel histories, the oldest fork point in one may not be in a pull request forked from the other.
- A list that reaches
--limitmay be incomplete, and a pull request left out of it may hold a trace the job would drop, so the job stops there instead. - A branch pushed with
github.tokenstarts no workflow run, and a pull request it opens starts none on its own, at most one that waits for approval, so the result can go unchecked. Give a GitHub App’s or a bot user’s token to bothactions/checkout, whose tokengit pushuses, andgh. - In a project with more than one schema, run
op compact --schema <name>once per schema, with that schema’s oplog at the frontier. Name a Seal with--sealto compact its sealed range too.
A branch the job cannot see, such as one never pushed, may still hold a dropped trace. The history checks fail its pull request when it opens.
Run the checks in GitLab CI
GitLab’s merges honour merge=union: two merge requests that each
append to an oplog merge without a conflict and leave a
fork. Nobody reconciles main. validate fails the
fork, so the history check in the pipeline that tests the merge result
keeps it out of main, and the fork is closed on the merge request’s
branch.
# .gitlab-ci.yml
oplog:
variables:
GIT_DEPTH: 0 # the whole history: the default of 20 commits refuses as shallow
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event" # the merge request's pipeline, a train's car
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH # branch pushes: the only pipelines with a before
before_script:
- curl -fsSL https://retrofit.sh/install | sh
- export PATH="$HOME/.local/bin:$PATH"
script:
- git fetch --quiet origin "$CI_DEFAULT_BRANCH"
- |
CANDIDATE=$CI_COMMIT_SHA
BEFORE=
if [ -n "${CI_MERGE_REQUEST_IID:-}" ]; then
TARGET=${CI_MERGE_REQUEST_TARGET_BRANCH_SHA:-$(git rev-parse FETCH_HEAD)}
elif [ "$CI_COMMIT_BRANCH" = "$CI_DEFAULT_BRANCH" ]; then
TARGET=$CI_COMMIT_BEFORE_SHA # main before this push
else
TARGET=$(git rev-parse FETCH_HEAD)
BEFORE=$CI_COMMIT_BEFORE_SHA # all zeros on a branch's first push
fi
git cat-file -e "$BEFORE^{commit}" 2>/dev/null || BEFORE=
export CANDIDATE TARGET BEFORE
sh ./ci/oplog-bundle.sh
if [ -n "${CI_MERGE_REQUEST_IID:-}" ]; then # the three-file step: merge request pipelines only
PR_HEAD=${CI_MERGE_REQUEST_SOURCE_BRANCH_SHA:-$CI_COMMIT_SHA}
export PR_HEAD
sh ./ci/oplog-three-file.sh
fi
The job runs on an image with git, curl and jq.
GIT_DEPTH: 0. A new GitLab project clones 20 commits by default, and the history checks need the whole history.- Branch pipelines.
CI_COMMIT_BEFORE_SHAis all zeros in every merge request pipeline, and only a branch pipeline started by a push has the push’s previous tip, which one compact-form check reads. So the job also runs on push pipelines; aworkflow:rulesthat drops duplicate pipelines must keep them. A scheduled or manual pipeline has no previous tip either, so the rule leaves it out. - What gates the merge. “Pipelines must succeed” checks only the merge request pipeline, so a branch pipeline’s failure of that check shows on the commit and does not block the merge. In the merge request pipeline, that check is reported unchecked.
- The target.
CI_MERGE_REQUEST_TARGET_BRANCH_SHAis set only in a merged results pipeline, a train’s car included; elsewhere the job fetches main for its tip. - The merge request’s head.
CI_MERGE_REQUEST_SOURCE_BRANCH_SHAis likewise set only in a merged results pipeline, whose commit is the temporary merge. In any other merge request pipeline, the commit the pipeline tests is the head itself. Main’s moving starts no merge request pipeline, so GitLab has no counterpart to GitHub’s job on main’s push: the merge request’s pipeline and a train’s car run the three-file step before the merge request merges.
For the annotations,
add the code quality conversion in after_script and name its file
under artifacts:reports:codequality, with artifacts:when: always so
a failed job still reports it.
Use a merge train, or semi-linear or fast-forward merges with “Pipelines must succeed” and automatic rebase off: the check gates what merges only where the pipeline tests the commit that merges. GitLab setups lists what each setup tests and what becomes of a fork on it.
A train cannot fix a car: a push to its branch drops it from the train. Close the fork on the branch before the train, with main in the branch’s history, so the next merge is trivial: bring main in and write the oplog from the three files, or close the fork in the one file where they are not at hand. After a rebase, use them too, and commit the result on top of the replayed commits. Two concurrent oplog merge requests in one train leave the second forked; it leaves the train, and every car behind it restarts. Add one oplog merge request to the train at a time to avoid the restarts.
On a setup that does not test what merges, more merge requests can
merge before main is fixed, and the fork grows a block with each.
reconcile closes a fork of three or more blocks two at a time, in line
order, on a branch; the result goes through the pipeline like any other
change.
Deploy and catch up a database
Deploy from a Seal: an unsealed operation’s position moves as the log grows (Sealed and unsealed ops). When a merge’s order is forced, the pull request’s operations are written before main’s, and a database deployed from main’s unsealed tail before that merge is left behind.
A database that takes only sealed operations catches up to the latest Seal from a dump of itself:
pg_dump --schema-only "$DATABASE_URL" > db.sql
retrofit migrate --from-dump db.sql --to seal > deploy.sql
psql -X -v ON_ERROR_STOP=1 -f deploy.sql "$DATABASE_URL"
A database deployed from the unsealed tail is a working copy. Check it
with identify, and where it reports the database behind, rebuild it
from the log, its rows saved first: render the whole history into a
fresh database, then move the rows across.
pg_dump --schema-only "$DATABASE_URL" > db.sql
retrofit identify db.sql # reports it behind, naming the operations after its nearest cut
retrofit migrate --from root --to last > rebuild.sql
psql -X -v ON_ERROR_STOP=1 -f rebuild.sql "$FRESH_DATABASE_URL"
Neither identify nor migrate --from-dump offers a catch-up for such
a database: the operations a forced merge put first do not replay over
a database that already holds main’s.
Related
- Check a pull request in CI: the steps these jobs run.
- History checks: each check, what each host and git path gives, and the GitLab setups that test what merges.
- Merge branches that change the oplog: resolve on the branch what these jobs report.
- Ship a hotfix ahead of the next release: deploy a fix ahead of the next release.