retrofit
0.14.0
retrofit.sh

Check a pull request in CI

A pull request’s check runs five steps over a project: the history checks over the commits the pull request brings, a merge of main’s tip with the pull request’s head, data check on every schema, the migration applied to a scratch database, and the findings turned into the host’s annotations. Each step below is a shell script a CI job runs at the project root, in a full clone, with retrofit and jq on its path. Gate merges on GitHub and GitLab wires the steps into each host’s jobs.

The steps read each schema’s oplog. Merge branches that change the oplog shows how a pull request brings main in and closes the fork a merge leaves.

Write and check the history bundle

retrofit validate --history <dir> runs the history checks over the commits an integration made. It runs no git, so the job first writes those commits into a directory, the history bundle (The bundle lists its members). The script reads three commits from its environment:

  • CANDIDATE: the commit the host tests, a pull request’s merge commit or a merge queue group’s commit.
  • TARGET: the target branch’s tip.
  • BEFORE: the push’s previous tip, when the host gives one. Leave it empty after a force push, when the old tip may not be in the clone. Without it, one compact-form check is reported unchecked.

Save the script in the repository, as ci/oplog-bundle.sh for one. It writes every member but the commits’ files and the optional holders, then runs validate --history until it names no file it lacks:

#!/bin/sh
# Write the history bundle for the commit under test, then check it.
# CANDIDATE and TARGET are required; BEFORE is optional; EVERY_COMMIT=1
# checks every commit of the pull request. Writes validate.ndjson.
set -eu
bundle=${BUNDLE:-$(mktemp -d)}
mkdir -p "$bundle/dead"
case "${BEFORE:-}" in *[!0]*) ;; *) BEFORE= ;; esac   # all zeros: no before

echo 1 > "$bundle/version"
git rev-parse --is-shallow-repository > "$bundle/shallow"
{
  echo "candidate=$CANDIDATE"
  echo "target=$TARGET"
  if [ -n "$BEFORE" ]; then echo "before=$BEFORE"; fi
} > "$bundle/roles"
git rev-list --parents "$CANDIDATE" "$TARGET" $BEFORE > "$bundle/graph"
git log --format='%H%n%B%x00' "$CANDIDATE" "$TARGET" $BEFORE > "$bundle/messages"

# The dead set, per schema: the target's tip, then the opid of every
# operation line (a ~ line and a Seal included) the schema's oplog held at
# any commit reachable from the target's tip.
prefix=$(git rev-parse --show-prefix)
retrofit schema list --as json > "$bundle.schemas"
jq -r 'select(.type == "item") | "\(.name) \(.path)"' "$bundle.schemas" |
while read -r name path; do
  {
    echo "$TARGET"
    git rev-list "$TARGET" | while read -r c; do
      if git cat-file -e "$c:$prefix$path" 2>/dev/null; then
        git cat-file blob "$c:$prefix$path"
      fi
    done | sed -n -E 's/^~?[A-Za-z]+\(([0-9a-f]{12}).*/\1/p' | sort -u
  } > "$bundle/dead/$name"
done

# Write each file validate names, and run it again until it names none.
while :; do
  status=0
  retrofit validate --history "$bundle" --allow-database-apply ${EVERY_COMMIT:+--every-commit} --as json \
    > validate.ndjson || status=$?
  jq -r '.data.history.missing[]?' validate.ndjson > "$bundle.missing"
  [ -s "$bundle.missing" ] || break
  while IFS= read -r m; do
    if [ -e "$bundle/$m" ] || [ -e "$bundle/$m.absent" ]; then
      echo "validate names $m again" >&2
      exit 2
    fi
    rest=${m#commits/}; sha=${rest%%/*}; file=${rest#*/}
    mkdir -p "$(dirname "$bundle/$m")"
    if git cat-file -e "$sha:$file" 2>/dev/null; then
      git cat-file blob "$sha:$file" > "$bundle/$m"
      git rev-parse "$sha:$file" > "$bundle/$m.blob"
    else
      : > "$bundle/$m.absent"
    fi
  done < "$bundle.missing"
done
exit "$status"

The first run names the files the checks need, and the script writes each one and runs again until none is named. It branches on that list, never on the exit, and the last run’s exit is the step’s. The step fails when a check fails, or when the bundle cannot be trusted, as from a shallow clone; History checks lists each check and what fails it.

--allow-database-apply states that CI applies the range to a database before it lands, which the scratch database step does on every push. With it, opaque-body lists a body the pull request adds that names no element, and passes. A pipeline without that step takes the flag out of the script, and the check then fails such a pull request.

The dead set (dead/<schema>) is the one member the script derives, from every commit reachable from the target’s tip, at the path the candidate’s config records. For a schema whose oplog moved, read each earlier path into its dead set as well.

Merge main’s tip with the pull request’s head

The history checks read only the files an integration wrote, so they cannot see a property main set and the pull request set again where no fork decides which value stands (What each path promises). This step runs the three-file merge over main’s tip and the pull request’s head, which stops on that conflict. Save it as ci/oplog-three-file.sh, for one:

#!/bin/sh
# Merge main's tip and the pull request's head with the three-file
# reconcile, per schema, and fail on a refusal. TARGET and PR_HEAD are
# required. Writes nothing in the project.
set -eu
t=$(mktemp -d)
base=$(git merge-base "$TARGET" "$PR_HEAD")
prefix=$(git rev-parse --show-prefix)
retrofit schema list --as json > "$t/schemas"
jq -r 'select(.type == "item") | "\(.name) \(.path)"' "$t/schemas" |
while read -r name path; do
  for c in "$base" "$TARGET" "$PR_HEAD"; do   # absent at this path on a side: skipped
    git cat-file -e "$c:$prefix$path" 2>/dev/null || continue 2
  done
  git show "$base:$prefix$path" > "$t/base.oplog"
  git show "$TARGET:$prefix$path" > "$t/main.oplog"
  git show "$PR_HEAD:$prefix$path" > "$t/pr.oplog"
  echo "$name:"
  retrofit reconcile --base "$t/base.oplog" --ours "$t/main.oplog" \
    --theirs "$t/pr.oplog" --result "$t/result.oplog"
done
  • TARGET: main’s tip, or, in a merge queue’s group or a train’s car, the commit the host merges the pull request onto.
  • PR_HEAD: the pull request’s head, the commit its author pushed, never the merge commit the host builds.

Each schema is read at the path the checkout’s config records, and a schema absent there on any side is skipped. For a schema whose oplog moved since the merge base, read that side’s earlier path instead.

The step fails when the merge stops. The job’s log then names the property, the operation each side set it with, and, for --keep and for --swap, the value the property ends as and the edit that undoes it. A stop other than a write-write is fork-forced-order: the merged order is forced and reports overrides nobody asked for. The log names the fork, why the order is forced and each override; the step fails until the merge is run with the flag that names the forced order.

Run the step on every push to the pull request, and where the host tests the merged result, on that: a merge queue’s group, a train’s car. Run it also in a job on main’s push, once for each open pull request that changes an oplog, with TARGET main’s new tip and PR_HEAD that pull request’s head (Check open pull requests when main moves). The step reads the pull request’s files as data and runs nothing from it.

Once a pull request has brought main in, the merge base holds main’s change, and the step has nothing to see. So bring main in with the three files, after a rebase as well as after a merge. Where the merge stops on a property both sides set, settle it as Settle a property both sides set shows; once that merge is committed, the step passes.

Check every commit of a pull request

By default only the candidate must validate. A commit between the fork point and the candidate may hold a fork, since a rebase replays each commit under union and reconcile runs at the tip. A team that merges by fast-forward puts every commit of the pull request on main. To have each one validate, under its own form line, set EVERY_COMMIT=1 on the bundle script, which passes --every-commit:

retrofit validate --history "$bundle" --allow-database-apply --every-commit --as json

Check the data files

data check passes a record it only lists, so the step reads the counts in the data-check record’s listed object. Fail the pull request on a listed record whose row shape migrate cannot apply, the two classes the script counts, since the deploy would stop on it. A record with no row-shape stamp is one migrate applies; never fail on it.

set -eu
retrofit schema list --as json > schemas.ndjson
jq -r 'select(.type == "item") | .name' schemas.ndjson |
while read -r name; do
  retrofit data check --schema "$name" --as json > data-check.ndjson
  jq -e 'select(.type == "data-check")
    | (.listed["anchor-shape-mismatch"] // 0) + (.listed["anchor-shape-unsupported"] // 0) == 0' \
    data-check.ndjson > /dev/null
done

A schema without a data root is an empty corpus, which data check passes, so the step runs on every project. A file data check cannot read fails the step on its own. The schemas are listed on a line of their own, so a schema list that fails over a retrofit.toml it cannot read stops the step rather than handing the loop no schemas. See retrofit data check for what each finding means.

Apply the migration to a scratch database

opaque-body lists a body a pull request adds that names no element, and no history check reads what such a body uses. One that uses a column another change removed, revised or dropped passes them, and fails where the migration is applied. A pull request that adds no body can break one main already holds, so run this step on every push. Apply the migration to a throwaway PostgreSQL in the pull request’s check (a service container the job starts) to find that, and any other body that does not apply. It shows the migration applies to an empty database, not that it applies to yours.

set -eu
retrofit schema list --as json > schemas.ndjson
jq -r 'select(.type == "item") | .name' schemas.ndjson |
while read -r name; do
  retrofit migrate --schema "$name" --from root --to last > "migrate-$name.sql"
  psql -X -v ON_ERROR_STOP=1 -f "migrate-$name.sql" "$SCRATCH_DATABASE_URL"
done

Each file is written before psql reads it, so a migrate that fails stops the step rather than handing psql an empty script. To rehearse the step a database deployed at the latest Seal takes, build the scratch database to the Seal first, then apply the rest, in place of the loop’s two lines:

retrofit migrate --schema "$name" --from root --to seal > to-seal.sql
retrofit migrate --schema "$name" --from seal --to last > after-seal.sql
psql -X -v ON_ERROR_STOP=1 -f to-seal.sql "$SCRATCH_DATABASE_URL"
psql -X -v ON_ERROR_STOP=1 -f after-seal.sql "$SCRATCH_DATABASE_URL"

On a schema with no Seal, --to seal fails; keep the loop’s two lines there. Applying with psql says why each psql line passes -X -v ON_ERROR_STOP=1.

Annotate the findings on GitHub or GitLab

validate --history --as json writes each failed check’s findings with the file and line each names (The record lists every field). An opaque-body finding also names the line’s operator, the body’s slot and, where one exists, its cause, beside the line and the rewrite. Under --allow-database-apply that check is advisory, so its findings stay in validate.ndjson and the filters below read none of them.

GitHub reads ::error lines from a step’s output as annotations:

jq -r '
  def esc: gsub("%"; "%25") | gsub("\r"; "%0D") | gsub("\n"; "%0A");
  (select(.type == "validate-history") | .data.history.checks[]
    | select(.status == "failed") | .check as $check | .findings[]
    | "::error file=\(.path),line=\(.line),title=\($check)::\(.message + (if .recovery then "; " + .recovery else "" end) | esc)"),
  (select(.type == "failed") | "::error title=\(.code)::\(.message + " (" + .hint + ")" | esc)")
' validate.ndjson

GitLab reads a code quality report, a JSON array the job names under artifacts:reports:codequality:

jq -s '[.[] | select(.type == "validate-history") | .data.history.checks[]
  | select(.status == "failed") | .check as $check | .findings[]
  | {description: .message, check_name: $check,
     fingerprint: "\($check):\(.commit):\(.path):\(.line):\(.opid // "")",
     severity: "major",
     location: {path: .path, lines: {begin: ([.line, 1] | max)}}}]' \
  validate.ndjson > gl-code-quality.json

When validate stops before any check runs, it names no line to annotate. On GitHub it is the second ::error line; on GitLab it is the job’s log.

Before 1.0, any release may change behavior; the release notes list what changed.

Published .