Maintainer and release engineering¶
Extra CODEOWNERS releases from main after CI succeeds. Maintainers review and
merge the change; they don't prepare a release pull request, edit a version
file, or create the normal release tag.
Pull-request checks¶
The CI workflow consolidates these build and test checks behind Required:
- Python formatting, linting, typing, workflow linting, and YAML linting
- the test suite on Python 3.12, 3.13, and 3.14, including PostgreSQL and backup/restore coverage on 3.12
- wheel and source-archive build plus a clean wheel installation
- strict MkDocs build and Markdown linting
- Helm lint, render, schema, security, and Kubernetes validation
- native
amd64andarm64container builds, smoke tests, and vulnerability scans.
The final Required job fails unless every group succeeds. Keep that stable
name in repository rules; matrix details can change without another branch
protection edit. Keep DCO, CodeQL, dependency review, the larger property-test
profile, and workflow-security checks as independent required controls. A
pull-request update cancels its stale run. Runs on main use queue: max, so a
later push cannot discard an earlier run.
Run the local subset with:
mise run check
That command exits zero when its checks pass. PostgreSQL and native cross-architecture behavior still need CI; the contributor guide explains how to run the complete database suite locally.
Automatic releases¶
On a push to main, the Release job invokes the reusable release workflow
after Required succeeds. The planner reads Git tags for that exact commit,
then refuses malformed versions, divergent release history, multiple tags on
one commit, and version collisions.
Confirm that GitHub's Immutable releases repository setting is enabled
before the next merge. The workflow's GITHUB_TOKEN cannot read that setting
in advance. Instead, the publisher stages a draft release after the other
artifacts are ready, uploads the assets, publishes the release, and requires
GitHub to report it as immutable. Only then does Release complete succeed.
Release-channel changes are explicit, append-only Git history. An ordinary
merge follows the latest release: an alpha increments its numeric suffix, while
a stable release applies conventional-commit rules to the commits since its
tag. To change that behavior, include one standard Git trailer in the commit
that lands on main:
Release-Channel: alpha
or:
Release-Channel: stable
The planner replays those transitions in commit order. alpha starts the next
semantic version as .alpha.1; stable promotes the current alpha base
without another conventional bump. A repeated, malformed, or duplicated
trailer fails the release rather than changing state implicitly. See the
contributor guide
for the merge-message procedure.
While the release channel is stable, commits since that tag select a semantic version bump:
BREAKING CHANGE,BREAKING-CHANGE, or a conventional!bumps majorfeatbumps minor- any other merged change bumps patch.
GitHub may start queued runs out of commit order. When a descendant run
publishes first, an older untagged run succeeds only after it proves that the
descendant tag is on main and the release is complete and immutable. It then
skips its own build and publication. This coalesces closely spaced merges into
one release instead of failing the older run or publishing versions out of
order.
The publisher refreshes main before each tag API write. If it observes a
newer commit, it defers publication. Its final job is Release deferred, not Release
complete, and its summary identifies the newer commit. No versioned image,
chart, or GitHub release is published by the deferred run. Native images may
already have been uploaded by digest for testing; those aren't a completed
release. These checks and GitHub's writes aren't atomic: if the tag reference
is successfully created, the run finishes that release even if main moves
while GitHub handles the request.
The newer push already has a queued CI run. It must pass the usual checks and publish before either change is released; the older run doesn't wait and block that queue. If the newer run fails, fix or rerun it. Don't deploy the version printed by a deferred plan. An existing tag still follows the partial-release recovery procedure below rather than being abandoned or moved.
That one calculated version feeds every artifact. The planner renders its
matching PEP 440 form for Hatch, the container receives the semantic version as
a build argument, and helm package overrides the chart's development
metadata. The checked-in chart stays at 0.0.0-dev.
Release jobs build the Python wheel, source distribution, Helm chart, and native images in parallel. The publisher then creates the immutable tag, assembles one multi-platform image, signs and attests the artifacts, pushes the OCI chart, and stages the GitHub release. It uploads the assets and publishes that release last. The release includes the Python artifacts; it does not publish them to PyPI.
Each image build exports its exact native filesystem without starting the
container and writes a raw per-platform distribution inventory. The published
release carries distribution-inventory-amd64.json and
distribution-inventory-arm64.json with their Sigstore bundles. Each inventory
is attested, signed, and bound to its matching child-image digest. The same
export produces recipient-notices-amd64.tar.gz and
recipient-notices-arm64.tar.gz; the publisher verifies them again after
artifact download and before allocating immutable release state. Read
recipient notice bundles for their exact
contents and limits. They preserve available notice material; they do not
decide that source or notice duties have been met. The inventory also records
the Debian distribution derived from the image's hashed canonical
/usr/lib/os-release file.
The native jobs also download the CPython source archive identified by the image's version and SHA-256. Each platform gets a signed, attested source bundle, verified again before the publisher allocates release state. A failed download or checksum mismatch stops publication. There is no additional version pin to update when Renovate changes the Python base.
After both native builds finish, a separate job collects one Debian source bundle for their combined package list. It retains version-matched source and Debian build recipes without rebuilding dependencies. Source archives use a SHA-256-checked download cache; cached bytes are checked again before use, and unused entries are removed. The cache is optional: missing or corrupt entries are downloaded again. Descriptors are fetched from Debian Snapshot on each run.
The publisher verifies debian-source.tar before allocating immutable release
state, then signs and attests it alongside the other assets. Source collection
fails if any installed Debian source package cannot be recovered or checked.
The remaining native-wheel and notice work is tracked in #18; the source bundles
do not claim exact upstream build provenance or legal approval.
Update a VEX claim¶
The reviewed source statement is security/vex/runtime.openvex.json. Use Vexcalibur as an
analysis aid when a vulnerability needs review, then check the affected package
URLs and impact statement in the pull request. The statement is a security
conclusion, not a way to hide a scanner result.
The VEX file also records hashes of the reviewed runtime inputs. CI rejects it
after a change to application files, uv.lock, pyproject.toml, Dockerfile,
or .dockerignore. Review the changed behavior against each existing claim
before refreshing the binding. Docs and tests do not require a refresh.
After that review, run this from the repository root with Python 3.12 or later.
Set REVIEW_RATIONALE to your explanation of the runtime changes and why the
claims still hold. The command updates the source VEX file; it does not change
a published release or decide whether a vulnerability is reachable.
python -I -S -B tools/release_vex.py bind-runtime \
--source security/vex/runtime.openvex.json --runtime-root . \
--review "$REVIEW_RATIONALE"
python -I -S -B tools/release_vex.py check-runtime \
--source security/vex/runtime.openvex.json --runtime-root .
The check succeeds silently when the recorded inputs match. Commit the updated statement with the runtime change so reviewers can inspect both. If a claim no longer holds, change or remove it before rebinding. Never add an automatic rebind step to CI.
When you add or change a vulnerability statement, give it a timestamp for
that assessment. Every statement needs an explicit date, so a new claim cannot
inherit an older document's issue time. The explicit bind command advances
the document revision and issue time when reviewed content changes, leaving
other statement dates alone. Repeating it with the same content and rationale
does not create another revision. The check also rejects document edits made
after binding. This bookkeeping does not perform the security review or choose
a vulnerability status.
The publisher copies the reviewed bytes only when every product URL matches the
signed native inventories. Debian URLs must identify the released architecture
and distribution. For Grype, use the inventory's distro_full identity, which
comes from DEBIAN_VERSION_FULL in the hashed os-release file, falling back
to VERSION_ID when that field is absent. An upstream qualifier must match the
installed package's source when present. Unknown qualifiers stop the release. This check
runs before the workflow creates an immutable tag or a versioned image
reference. The publisher then signs that release asset and attaches an OpenVEX
attestation to the final multi-platform image digest. If a package update or a
change in reachable behavior invalidates a conclusion, update or remove the
source statement and release the correction. Never try to rewrite a published
VEX asset.
The runtime review
records the package versions, native-library checks, and limits of the current
assessment. under_investigation is not an exemption: the finding still blocks
the High/Critical gate. The older OpenSSL-only files remain as historical
evidence; current builds use the combined runtime statement.
The October 5 runtime update also moves CPython from 3.14.7 to 3.14.8; review
its compatibility tests and source evidence along with the Debian packages.
Before allocating another version, the workflow requires the preceding release
to be published and immutable. It accepts v0.1.0-alpha.7 as the sole
historical exception because that public release predates this policy. No later
mutable release can advance the version chain.
If publication stops halfway through, rerun the failed jobs in the original CI run. GitHub keeps that run on its original commit. A tag already on that commit is reused, and a completed GitHub release switches the workflow into verification mode. It verifies the required assets, both image architectures, and the exact image and chart digests. It then verifies GitHub provenance and Sigstore evidence against the release workflow and original commit, including the raw inventory's platform-digest binding and the OpenVEX statement's image binding. Conflicting, missing, or ambiguous evidence stops the retry.
If the postcondition reports a mutable release, first enable immutable releases and query that release again. If GitHub still reports it as mutable, delete only that incomplete GitHub release. Keep its tag, image, and chart, then rerun the failed jobs in the original CI run so the workflow can verify and reuse them.
The retry is complete only when its Release complete job succeeds. Don't
dispatch a fresh CI run for a newer commit as a substitute, and don't move or
delete a tag to clear failed verification.
The scheduled Cold container build workflow builds both architectures with
BuildKit caching disabled. Treat its failure as evidence that the normal cache
hid a missing or mutable input.
Maintainer procedures¶
- Review current project status
- Read the decision records
- Review the runtime base image
- Respond to a dependency audit
- Rescan a published image
- Review or recover a Renovate update
- Review stacked pull requests
- Run the live GitHub contract
- Read live GitHub evidence reports
- Update the tutorial webhook relay
- Review the DCO evidence contract