Skip to content

Maintainer and release engineering

Extra CODEOWNERS releases from main after CI succeeds. Maintainers review and merge the change; they don't prepare a release pull request, edit a version file, or create the normal release tag.

Pull-request checks

The CI workflow consolidates these build and test checks behind Required:

  • Python formatting, linting, typing, workflow linting, and YAML linting
  • the test suite on Python 3.12, 3.13, and 3.14, including PostgreSQL and backup/restore coverage on 3.12
  • wheel and source-archive build plus a clean wheel installation
  • strict MkDocs build and Markdown linting
  • Helm lint, render, schema, security, and Kubernetes validation
  • native amd64 and arm64 container builds, smoke tests, and vulnerability scans.

The final Required job fails unless every group succeeds. Keep that stable name in repository rules; matrix details can change without another branch protection edit. Keep DCO, CodeQL, dependency review, the larger property-test profile, and workflow-security checks as independent required controls. A pull-request update cancels its stale run. Runs on main use queue: max, so a later push cannot discard an earlier run.

Run the local subset with:

mise run check

That command exits zero when its checks pass. PostgreSQL and native cross-architecture behavior still need CI; the contributor guide explains how to run the complete database suite locally.

Automatic releases

On a push to main, the Release job invokes the reusable release workflow after Required succeeds. The planner reads Git tags for that exact commit, then refuses malformed versions, divergent release history, multiple tags on one commit, and version collisions.

Confirm that GitHub's Immutable releases repository setting is enabled before the next merge. The workflow's GITHUB_TOKEN cannot read that setting in advance. Instead, the publisher stages a draft release after the other artifacts are ready, uploads the assets, publishes the release, and requires GitHub to report it as immutable. Only then does Release complete succeed.

Release-channel changes are explicit, append-only Git history. An ordinary merge follows the latest release: an alpha increments its numeric suffix, while a stable release applies conventional-commit rules to the commits since its tag. To change that behavior, include one standard Git trailer in the commit that lands on main:

Release-Channel: alpha

or:

Release-Channel: stable

The planner replays those transitions in commit order. alpha starts the next semantic version as .alpha.1; stable promotes the current alpha base without another conventional bump. A repeated, malformed, or duplicated trailer fails the release rather than changing state implicitly. See the contributor guide for the merge-message procedure.

While the release channel is stable, commits since that tag select a semantic version bump:

  • BREAKING CHANGE, BREAKING-CHANGE, or a conventional ! bumps major
  • feat bumps minor
  • any other merged change bumps patch.

GitHub may start queued runs out of commit order. When a descendant run publishes first, an older untagged run succeeds only after it proves that the descendant tag is on main and the release is complete and immutable. It then skips its own build and publication. This coalesces closely spaced merges into one release instead of failing the older run or publishing versions out of order.

The publisher refreshes main before each tag API write. If it observes a newer commit, it defers publication. Its final job is Release deferred, not Release complete, and its summary identifies the newer commit. No versioned image, chart, or GitHub release is published by the deferred run. Native images may already have been uploaded by digest for testing; those aren't a completed release. These checks and GitHub's writes aren't atomic: if the tag reference is successfully created, the run finishes that release even if main moves while GitHub handles the request.

The newer push already has a queued CI run. It must pass the usual checks and publish before either change is released; the older run doesn't wait and block that queue. If the newer run fails, fix or rerun it. Don't deploy the version printed by a deferred plan. An existing tag still follows the partial-release recovery procedure below rather than being abandoned or moved.

That one calculated version feeds every artifact. The planner renders its matching PEP 440 form for Hatch, the container receives the semantic version as a build argument, and helm package overrides the chart's development metadata. The checked-in chart stays at 0.0.0-dev.

Release jobs build the Python wheel, source distribution, Helm chart, and native images in parallel. The publisher then creates the immutable tag, assembles one multi-platform image, signs and attests the artifacts, pushes the OCI chart, and stages the GitHub release. It uploads the assets and publishes that release last. The release includes the Python artifacts; it does not publish them to PyPI.

Each image build exports its exact native filesystem without starting the container and writes a raw per-platform distribution inventory. The published release carries distribution-inventory-amd64.json and distribution-inventory-arm64.json with their Sigstore bundles. Each inventory is attested, signed, and bound to its matching child-image digest. The same export produces recipient-notices-amd64.tar.gz and recipient-notices-arm64.tar.gz; the publisher verifies them again after artifact download and before allocating immutable release state. Read recipient notice bundles for their exact contents and limits. They preserve available notice material; they do not decide that source or notice duties have been met. The inventory also records the Debian distribution derived from the image's hashed canonical /usr/lib/os-release file.

The native jobs also download the CPython source archive identified by the image's version and SHA-256. Each platform gets a signed, attested source bundle, verified again before the publisher allocates release state. A failed download or checksum mismatch stops publication. There is no additional version pin to update when Renovate changes the Python base.

After both native builds finish, a separate job collects one Debian source bundle for their combined package list. It retains version-matched source and Debian build recipes without rebuilding dependencies. Source archives use a SHA-256-checked download cache; cached bytes are checked again before use, and unused entries are removed. The cache is optional: missing or corrupt entries are downloaded again. Descriptors are fetched from Debian Snapshot on each run.

The publisher verifies debian-source.tar before allocating immutable release state, then signs and attests it alongside the other assets. Source collection fails if any installed Debian source package cannot be recovered or checked. The remaining native-wheel and notice work is tracked in #18; the source bundles do not claim exact upstream build provenance or legal approval.

Update a VEX claim

The reviewed source statement is security/vex/runtime.openvex.json. Use Vexcalibur as an analysis aid when a vulnerability needs review, then check the affected package URLs and impact statement in the pull request. The statement is a security conclusion, not a way to hide a scanner result.

The VEX file also records hashes of the reviewed runtime inputs. CI rejects it after a change to application files, uv.lock, pyproject.toml, Dockerfile, or .dockerignore. Review the changed behavior against each existing claim before refreshing the binding. Docs and tests do not require a refresh.

After that review, run this from the repository root with Python 3.12 or later. Set REVIEW_RATIONALE to your explanation of the runtime changes and why the claims still hold. The command updates the source VEX file; it does not change a published release or decide whether a vulnerability is reachable.

python -I -S -B tools/release_vex.py bind-runtime \
  --source security/vex/runtime.openvex.json --runtime-root . \
  --review "$REVIEW_RATIONALE"
python -I -S -B tools/release_vex.py check-runtime \
  --source security/vex/runtime.openvex.json --runtime-root .

The check succeeds silently when the recorded inputs match. Commit the updated statement with the runtime change so reviewers can inspect both. If a claim no longer holds, change or remove it before rebinding. Never add an automatic rebind step to CI.

When you add or change a vulnerability statement, give it a timestamp for that assessment. Every statement needs an explicit date, so a new claim cannot inherit an older document's issue time. The explicit bind command advances the document revision and issue time when reviewed content changes, leaving other statement dates alone. Repeating it with the same content and rationale does not create another revision. The check also rejects document edits made after binding. This bookkeeping does not perform the security review or choose a vulnerability status.

The publisher copies the reviewed bytes only when every product URL matches the signed native inventories. Debian URLs must identify the released architecture and distribution. For Grype, use the inventory's distro_full identity, which comes from DEBIAN_VERSION_FULL in the hashed os-release file, falling back to VERSION_ID when that field is absent. An upstream qualifier must match the installed package's source when present. Unknown qualifiers stop the release. This check runs before the workflow creates an immutable tag or a versioned image reference. The publisher then signs that release asset and attaches an OpenVEX attestation to the final multi-platform image digest. If a package update or a change in reachable behavior invalidates a conclusion, update or remove the source statement and release the correction. Never try to rewrite a published VEX asset.

The runtime review records the package versions, native-library checks, and limits of the current assessment. under_investigation is not an exemption: the finding still blocks the High/Critical gate. The older OpenSSL-only files remain as historical evidence; current builds use the combined runtime statement. The October 5 runtime update also moves CPython from 3.14.7 to 3.14.8; review its compatibility tests and source evidence along with the Debian packages.

Before allocating another version, the workflow requires the preceding release to be published and immutable. It accepts v0.1.0-alpha.7 as the sole historical exception because that public release predates this policy. No later mutable release can advance the version chain.

If publication stops halfway through, rerun the failed jobs in the original CI run. GitHub keeps that run on its original commit. A tag already on that commit is reused, and a completed GitHub release switches the workflow into verification mode. It verifies the required assets, both image architectures, and the exact image and chart digests. It then verifies GitHub provenance and Sigstore evidence against the release workflow and original commit, including the raw inventory's platform-digest binding and the OpenVEX statement's image binding. Conflicting, missing, or ambiguous evidence stops the retry.

If the postcondition reports a mutable release, first enable immutable releases and query that release again. If GitHub still reports it as mutable, delete only that incomplete GitHub release. Keep its tag, image, and chart, then rerun the failed jobs in the original CI run so the workflow can verify and reuse them.

The retry is complete only when its Release complete job succeeds. Don't dispatch a fresh CI run for a newer commit as a substitute, and don't move or delete a tag to clear failed verification.

The scheduled Cold container build workflow builds both architectures with BuildKit caching disabled. Treat its failure as evidence that the normal cache hid a missing or mutable input.

Maintainer procedures