Deploy an alpha in shadow mode¶
Use this guide to deploy the self-hosted GitHub App with PostgreSQL and the published Helm chart. The current releases are alpha software. Run the check as non-required evidence while you test it; don't let it authorize a production merge.
Danger
Keep GitHub's native Require review from Code Owners rule on production
repositories. The commit-scoped Check Run behavior still has an open
provider contract in
issue #1. Do not
deploy, mirror, or redistribute legacy preview images: :main and its
sha-* and sha256-* companion tags. They are unsupported and unsafe for
deployment because they predate the immutable release pipeline.
Choose one release¶
On a push to main, the release job runs after required CI succeeds and
publishes one immutable release. Use the same version everywhere:
- container image:
ghcr.io/stampbot/extra-codeowners:VERSION - Helm chart:
oci://ghcr.io/stampbot/charts/extra-codeowners - Python wheel, source distribution, signatures, and scan reports: the matching GitHub release.
You don't need the raw per-platform distribution inventories to deploy the App. New releases include them in the matching GitHub release for component review. They are not a notice package or a source offer; see project status for their scope and limits.
If the release contains a reviewed VEX claim, the same release includes
extra-codeowners-VERSION.openvex.json and its Sigstore bundle. Its OpenVEX
attestation binds that statement to the multi-platform image digest. Use that
asset with the released digest, not a statement from a newer source checkout.
Replace VERSION with an exact value such as 0.1.0-alpha.8. Don't use a
floating tag. Resolve and record the multi-platform image digest before you
change the cluster; the chart also accepts that digest through image.digest.
The released chart sets its appVersion to the same version and uses it as the
default image tag. Confirm that before installation:
export VERSION='0.1.0-alpha.8'
helm show chart \
oci://ghcr.io/stampbot/charts/extra-codeowners \
--version "$VERSION"
The output must show version and appVersion equal to VERSION. Check the
matching GitHub release before continuing. The release workflow signs and
attests the image, chart, and Python artifacts, but verification is still the
operator's job. A workflow file by itself does not prove that publication
finished.
Require GitHub to report the release as published and immutable:
test "$(
gh api "repos/stampbot/extra-codeowners/releases/tags/v$VERSION" \
--jq '.draft == false and .immutable == true'
)" = true
The command exits silently on success. Stop if it prints an error or exits nonzero.
Resolve the image digest and verify that GitHub recorded provenance from this repository's release workflow:
export IMAGE_REPOSITORY='ghcr.io/stampbot/extra-codeowners'
export IMAGE_DIGEST="$(
docker buildx imagetools inspect "$IMAGE_REPOSITORY:$VERSION" |
awk '$1 == "Digest:" {print $2; exit}'
)"
printf '%s\n' "$IMAGE_DIGEST" | grep -Eq '^sha256:[0-9a-f]{64}$'
gh attestation verify "oci://$IMAGE_REPOSITORY@$IMAGE_DIGEST" \
--repo stampbot/extra-codeowners \
--signer-workflow stampbot/extra-codeowners/.github/workflows/release.yml
Every command must exit zero. Keep IMAGE_DIGEST in the deployment record and
pass it as the chart's image.digest. The README.md inside the matching
chart archive contains the chart verification and install commands.
For a source review or local image test, the Dockerfile installs the runtime
graph from uv.lock and rejects dependency source builds. Release builds run
natively on amd64 and arm64, scan each image, and assemble the
multi-platform tag only after both builds pass. The Dockerfile
pins the exact base image by digest, and the
runtime-base explanation covers the choice and
its residual risk.
Prerequisites¶
Before scheduling a deployment, obtain:
- an Extra CODEOWNERS alpha image, verified and pinned by platform digest
- the exact chart source associated with that image
- a GitHub App with the required permissions and events
- a public HTTPS origin with a valid certificate
- PostgreSQL that accepts TLS with
sslmode=require, or an operator-controlled local proxy or Unix socket - tested PostgreSQL backups and a database credential restricted to this service
- a secret manager for the App private key and webhook secret
- outbound HTTPS access to the configured GitHub API
- reliable UTC clock synchronization on every node
- access to the health endpoints, logs, and Prometheus metrics
- the Helm 3.x version pinned in
the repository's
mise.toml, Docker Buildx, and a current GitHub CLI on the administration host.
Clock accuracy matters because GitHub App JSON Web Tokens, setup-state expiry, and database leases use wall-clock time.
Use SQLite only for a single-process development installation. It is not a production queue.
Provision PostgreSQL¶
Create one database and one role for Extra CODEOWNERS. Let that role own only the application database; don't grant PostgreSQL cluster administration or access to unrelated databases.
Use a SQLAlchemy URL through the exact postgresql+psycopg driver:
postgresql+psycopg://DB_USER:DB_PASSWORD@DB_HOST:5432/DB_NAME?sslmode=require
Replace every uppercase placeholder. The host, database, username, and nonempty password must all appear in the URL. Percent-encode reserved characters in the username and password, then treat the complete URL as a secret. An explicit port must be between 1 and 65535; omit it only when port 5432 is correct.
Use sslmode=require for a remote database. It encrypts the connection but
does not verify the database certificate or hostname. The application prevents
libpq from loading its default root certificate, so a file in the process
account does not change that behavior. Keep the database route on a private
network you control and restrict the database role to this service.
Use one route. A query-string host may supply the host only when the URL
authority omits it. An explicit hostaddr also requires that host and
sslmode=require, even when the host looks local. Only localhost,
127.0.0.1, ::1, or a Unix-socket path may use an operator-controlled local
transport without TLS. Hostless and comma-separated routes are rejected. You
must authenticate and secure a local proxy's upstream connection.
Only host, hostaddr, and sslmode query parameters are supported.
Connection-service URLs, PGSERVICE, PGSERVICEFILE, .pgpass, and
PGPASSFILE are not. Keep every ambient libpq connection variable out of the
application and migrator environments; production validation fails even when
one is present with an empty value. Both processes disable GSSAPI encryption
so it cannot bypass the required SSL transport, and pin search_path=public.
Use sslmode=require or sslmode=disable; certificate-verification modes are
unsupported.
Normal service startup checks the Alembic head and the
required-release-contract. It never creates or changes schema. Follow
Upgrade, back up, and restore before the first startup and every
upgrade.
Database operations have fixed fail-fast limits:
| Operation | Limit |
|---|---|
| Connect to PostgreSQL | 3 seconds |
| Obtain an application-pool connection | 2 seconds |
| Run an ordinary statement | 3 seconds |
| Wait for the default migration advisory lock | 60 seconds |
| Run one migration statement | 60 seconds |
The Helm migration Job has separate Secret, environment, volume, mount, and ServiceAccount settings. Give it the database URL and only the authority needed to change this schema. Its Secret must not contain ambient libpq variables. Never mount the GitHub private key or webhook secret into the migration Job.
Test the complete application-to-database path, including any proxy, under the expected peak latency and concurrency. If normal operations approach these limits, don't use the service as merge infrastructure. A timeout blocks or retries work; it never infers approval.
Mount credentials¶
Don't bake credentials into the image, pass them as command-line arguments, or commit them to an environment file.
Mount the private key and webhook secret as read-only files that only the container's runtime user can read. Configure:
EXTRA_CODEOWNERS_GITHUB_PRIVATE_KEY_FILE=/run/secrets/github-private-key
EXTRA_CODEOWNERS_GITHUB_WEBHOOK_SECRET_FILE=/run/secrets/github-webhook-secret
Inject the database URL through the platform's secret mechanism. Don't add
libpq PG* connection variables to the same process environment.
Configure the process¶
Set at least these values:
EXTRA_CODEOWNERS_ENVIRONMENT=production
EXTRA_CODEOWNERS_GITHUB_APP_ID=123456
EXTRA_CODEOWNERS_GITHUB_PRIVATE_KEY_FILE=/run/secrets/github-private-key
EXTRA_CODEOWNERS_GITHUB_WEBHOOK_SECRET_FILE=/run/secrets/github-webhook-secret
EXTRA_CODEOWNERS_DATABASE_URL=postgresql+psycopg://DB_USER:DB_PASSWORD@DB_HOST:5432/DB_NAME?sslmode=require
EXTRA_CODEOWNERS_WORKER_RETRY_MAX_SECONDS=60
EXTRA_CODEOWNERS_WEBHOOK_DELIVERY_RETENTION_DAYS=30
Replace the App ID and database placeholders. Keep
EXTRA_CODEOWNERS_ALLOW_INSECURE_CHANGES=false.
Setup mode is disabled by default. Keep it off after App registration. Set
EXTRA_CODEOWNERS_PUBLIC_URL only for a separate setup process, and use an
HTTPS origin.
Production startup rejects:
- SQLite and every URL that does not use the exact
postgresql+psycopgdriver - a URL without one explicit host, database, username, and nonempty password
- remote PostgreSQL without
sslmode=require - connection-service URLs, unknown query parameters, and ambient libpq connection variables
- a webhook secret shorter than 32 UTF-8 bytes
- a non-HTTPS GitHub API origin
- missing App ID, private key, or webhook secret.
Choose a webhook delivery-ID retention period that covers GitHub redelivery and your incident investigation window. The elected reconciler removes expired IDs. Keep reconciliation enabled or create a separately reviewed pruning procedure. Once an ID expires, GitHub may redeliver it and the service may accept it again, but the worker still fetches current GitHub evidence.
Evaluation and authority failures retry indefinitely. The
EXTRA_CODEOWNERS_WORKER_RETRY_MAX_SECONDS setting caps the ordinary
exponential delay, not the attempt count. GitHub rate-limit responses use a
separate provider-supplied delay bounded to one day. Alert on a queue that
stays above its normal baseline and on repeated failure logs. Don't use manual
requeue as routine recovery.
See the runtime settings reference for every setting, default, and bound.
Size the Kubernetes startup probe¶
The Helm chart enables a startup probe on /health/live. It runs every five
seconds, waits up to three seconds for each response, and allows 60 failures.
That gives initialization a five-minute budget before Kubernetes restarts the
container.
Keep the startup probe separate from the other two probes. Kubernetes doesn't run liveness or readiness probes until startup succeeds. After that point, liveness detects a stuck process or failed local task, while readiness removes an instance from webhook traffic when its database, App identity, worker, or reconciler is unavailable.
Measure initialization with the production database path and secret mounts.
Set periodSeconds * failureThreshold above the slowest expected startup plus
operational margin, and set timeoutSeconds above the normal /health/live
response time. Keep Helm and rollout wait timeouts above that startup budget.
Increasing the startup allowance does not weaken ongoing liveness or readiness
checks.
Expose only the webhook¶
Route unauthenticated GitHub traffic only to POST /webhooks/github. The
proxy must preserve the raw body and these headers:
X-Hub-Signature-256X-GitHub-EventX-GitHub-Delivery.
Use current TLS protocols, disable response caching, and reject request bodies larger than 10 MiB before buffering. Use a lower limit only after testing real delivery sizes. Rate limits must still accommodate normal GitHub bursts and manual redelivery.
Disable query-string logging for /setup/callback. Its one-use App
Manifest conversion code is sensitive. With setup disabled, /setup returns
404. A callback request that includes
its required code and state parameters also returns 404.
Use path-level access control even when every route shares one HTTPS origin.
Keep /, /api/runtime-identity, /api/docs, /api/openapi.json,
/docs/oauth2-redirect, /metrics, both /health routes, and every /setup
route behind operator-controlled routing. If the proxy cannot enforce that
split, require network or proxy authentication everywhere except the exact
webhook path.
Verify the deployment¶
Run these checks from the orchestrator network. Replace the example hostname with an operator-only endpoint:
curl --fail-with-body https://operator-endpoint.example.com/health/live
curl --fail-with-body https://operator-endpoint.example.com/health/ready
curl --fail-with-body \
https://operator-endpoint.example.com/api/runtime-identity
All three requests must return HTTP 200. Compare every runtime-identity field
with the reviewed deployment settings. An official image must report its full
verified source commit in build_revision; a source installation reports
null. The endpoint is a deployment self-report, not independent provenance
evidence.
If the instance runs background work, confirm that worker_enabled,
reconciler_enabled, worker, and reconciler are true in both health
responses. The first two fields confirm that the process is configured to run
those tasks. The other two report task health.
Confirm the metrics scraper can read /metrics and that
extra_codeowners_insecure_changes_enabled is 0.
After reconciliation runs, verify that at least one scraped replica reports an
increase in extra_codeowners_reconciliations_total{result="success"}. At
least one replica's
extra_codeowners_reconciliation_last_success_timestamp_seconds must contain a
recent timestamp. Investigate any partial or failure result before treating
reconciliation as healthy.
Then use a disposable repository with test policy:
- Open a pull request that changes an owned path.
- In the App's Advanced settings, confirm a successful
pull_request.openeddelivery. - Confirm that the expected App publishes the check on the current head.
- Redeliver the same delivery and confirm it does not create duplicate work.
- Complete every negative test in Prepare repository rules.
Passing these checks does not close the commit-scoped Check Run limitation. Do not use the result to authorize production merges.
Roll back or mitigate¶
Restore repository enforcement before changing traffic or processes:
- Restore GitHub's native Require review from Code Owners rule on every affected repository.
- Wait until GitHub shows the rule as active.
- Remove the
Extra CODEOWNERS / approvalrequirement if the service's decisions are suspect or the service will remain offline. - Stop routing new webhook traffic.
This order applies to an authorization defect, a same-schema application rollback, and a database restore. It prevents an earlier Extra CODEOWNERS success from remaining the only code-owner gate while the service is stopped.
If you have a previously verified compatible image, compare the current database head with that artifact's required head:
- If the head is unchanged, deploy the previous image by its recorded digest
and run
extra-codeowners database checkbefore restoring traffic. - If the head changed, stop every Extra CODEOWNERS process, preserve the failed database, and restore the verified pre-migration backup into a new empty database. Validate the restored database with the previous artifact.
- Redeliver deliveries that the service never accepted. Let pending work retry, and let reconciliation enqueue open pull requests with no job.
- Verify current-head checks in a disposable repository.
Every Alembic head change requires the restore in step 2. An additive physical change does not make an old exact-head artifact compatible.
If you don't have a previously verified compatible image, keep native code-owner enforcement in place and preserve the database and sanitized logs for investigation.
The Helm chart runs a bounded pre-upgrade migration Job. The default Deployment
strategy is still Recreate, but the reviewed two-replica preset uses a
RollingUpdate. A rolling workload protects a pod or node failure; it does
not make an old application compatible with a new Alembic head. Before any
release whose upgrade notes require a drain, route webhook traffic away and
scale the old Deployment to zero. An HPA or GitOps controller can undo that
scale operation, so suspend reconciliation and remove the HPA first. Follow
the controller-safe drain procedure,
wait for termination and both the worker and reconciler lease periods, and
then start the upgrade.
Expect a webhook interruption. GitHub does not automatically redeliver failed deliveries, so inspect and redeliver them after readiness returns. Reconciliation output is advisory; independently inventory accessible open pull requests and retain native enforcement for any current check you did not verify.
For a PostgreSQL deployment, enable the chart's highAvailability preset once
you have two schedulable nodes. The database coordinates competing claims and
the singleton reconciler lease across replicas. Use
charts/extra-codeowners/README.md from the same reviewed checkout; don't
jump to a mutable default-branch copy.
What a release publishes¶
Each successful release workflow publishes:
- a signed multi-architecture image at
ghcr.io/stampbot/extra-codeowners:VERSION - a signed OCI chart at
oci://ghcr.io/stampbot/charts/extra-codeowners - Python wheel and source distribution as GitHub release assets
- signed raw distribution inventories for
amd64andarm64, each bound to its image child digest - a signed OpenVEX release asset, bound to the multi-platform image digest
- BuildKit provenance and software bill of materials for each native image
- GitHub provenance attestations and Sigstore signatures for published artifacts
- native vulnerability reports.
A published, immutable GitHub release is the pipeline's completion record. Verify the release, GHCR image, and OCI chart before using them. The Python artifacts are not published to the Python Package Index (PyPI).
Environment-specific chart upgrade evidence and a reproducible Google Cloud deployment guide are still planned. Their workload-identity behavior will be documented from published artifacts, not inferred from source.