> ## Documentation Index > Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt > Use this file to discover all available pages before exploring further. # Provenance bundle signing (cosign) > W10.1 + #27 — sign /api/v1/decisions/:id/provenance bundles with sigstore cosign. Spawn-based, env-var-keyed, fail-soft to unsigned when COSIGN_KEY is unset or the binary is missing. ## What it does Every call to `GET /api/v1/decisions/:id/provenance` returns a canonicalised provenance bundle (decision trace + model snapshots + audit chain + SLSA attestation + fairness slice). When the deployment is configured for sigstore signing, the bundle is also signed with `cosign sign-blob` and the detached signature lands in the response header: ``` X-Provenance-Digest: 7c3e… (sha256 of the canonical bundle) X-Provenance-Signature: MEUCIQDk… (cosign sign-blob output) X-Provenance-Schema: kaireon.provenance.v1 ``` When signing is not configured, the same bundle is returned with: ``` X-Provenance-Signature: unsigned ``` so consumers can distinguish a key-bound bundle from a dev / unsigned bundle without re-parsing the body. ## Configuration Env vars on the runtime container: | Env var | Required | Meaning | | ----------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **COSIGN\_KEY** | yes (for signing) | Cosign-format private key bytes (e.g., the contents of `cosign.key` your operator generated with `cosign generate-key-pair`). Cosign accepts the raw bytes via the `env://` URI scheme — see below. | | `COSIGN_BINARY` | no | Override the binary path (default: `cosign` on PATH). The image installs cosign at `/usr/local/bin/cosign`. | | `COSIGN_PASSWORD` | only for passphrase-protected keys | Read natively by cosign to decrypt an encrypted private key. The platform never references it directly — the child process inherits the container environment. | Optional sigstore-transparency env vars (Rekor upload, Fulcio OIDC) are honoured but not required — the V1 wire ships **offline-key signing** only. Keyless / Fulcio / OIDC support is a follow-up. ## How the signing works `lib/supply-chain/cosign-sign-blob.ts` invokes: ``` cosign sign-blob \ --key=env://COSIGN_KEY \ --tlog-upload=false \ --yes \ - # blob payload via stdin ``` The signature is read from the child's **stdout** — the code deliberately does **not** pass `--output-signature` (all three variants of it fail in the container: `-` becomes a literal filename on a read-only cwd, and `/dev/stdout` / `/proc/self/fd/1` raise ENXIO under Alpine/Go). `--tlog-upload=false` skips the Sigstore Rekor transparency log: the V1 wire is offline-key signing; operators who want tlog can enable it out-of-band via cosign's own env vars. Three discipline rules are enforced by code: 1. **`spawn` only — never `exec`.** No shell is invoked, so there is no command-injection surface. The full argv is a fixed array of constants. 2. **The signing key never appears on argv.** Cosign reads the key bytes from the env var named by the `env://COSIGN_KEY` URI; only the URI string lives in argv. 3. **The blob payload arrives via stdin.** No tempfile, no argv leak. The canonical-JSON bundle is piped to the child's stdin and the stream is closed immediately. The argv is a fixed array of constants — the key bytes reach cosign only through the inherited process environment (`env://COSIGN_KEY`), never through arguments. ## Failure modes (fail-soft to "unsigned") | Condition | Header value | Audit log changes field | | ------------------------------------------------------------ | ------------ | --------------------------------------------------------- | | **COSIGN\_KEY** env unset | `unsigned` | `signed: false` (no `signatureFailureCode`) | | Cosign binary missing on PATH | `unsigned` | `signed: false`, `signatureFailureCode: "binary_missing"` | | Cosign exited non-zero | `unsigned` | `signed: false`, `signatureFailureCode: "exit_nonzero"` | | Cosign exited 0 but with empty stdout | `unsigned` | `signed: false`, `signatureFailureCode: "exit_nonzero"` | | 30s spawn timeout | `unsigned` | `signed: false`, `signatureFailureCode: "timeout"` | | spawn-time OS error (permission denied, out of memory, etc.) | `unsigned` | `signed: false`, `signatureFailureCode: "spawn_error"` | Stderr from cosign is captured (capped at 8 KB) and logged to the application logs (the first 256 chars, via the structured logger) for operator debugging. The audit row records the typed `signatureFailureCode` only — cosign diagnostics never reach the audit log, the response body, or the response headers. ## Image layout The runtime stage of `platform/Dockerfile` installs cosign: ```dockerfile theme={null} ARG COSIGN_VERSION=v2.4.1 RUN apk add --no-cache curl ca-certificates && \ arch="$(uname -m)" && \ case "$arch" in \ x86_64) cosign_arch="amd64";; \ aarch64) cosign_arch="arm64";; \ *) echo "unsupported arch: $arch"; exit 1;; \ esac && \ curl --fail --silent --show-error --location \ --output /usr/local/bin/cosign \ "https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-${cosign_arch}" && \ chmod 0755 /usr/local/bin/cosign && \ /usr/local/bin/cosign version && \ apk del curl ``` We deliberately do **not** hard-code a sha256 in the Dockerfile because any value baked in here would be unverifiable without network access during the review. The cosign release artefacts are themselves sigstore-signed; the trustworthy verification path is to run `cosign verify-blob` against the sigstore root after image build, as a separate CI step. That bootstrap is operator-authorised follow-up. ## Installing cosign signing Bundles are returned **unsigned** until you provide a signing key. There are exactly two supported install paths — pick whichever matches your deployment topology. Both paths set the same two env vars (**COSIGN\_KEY**, **COSIGN\_PASSWORD**) on the runtime container; what differs is where those values live at rest. ### Path A — Cloud (AWS Secrets Manager) Use this on App Runner / ECS / EKS / any AWS-resident deployment. Keys live encrypted in Secrets Manager and the runtime resolves them at container start. Steps: 1. `cosign generate-key-pair` (one time) → `cosign.key` + `cosign.pub`. 2. `aws secretsmanager create-secret --name /cosign-key --secret-string file://cosign.key` 3. `aws secretsmanager create-secret --name /cosign-password --secret-string ''` 4. Grant the runtime instance role `secretsmanager:GetSecretValue` on those two ARNs (least-privilege; one inline policy is enough). 5. Wire the ARNs as **runtime secrets** (App Runner runtime-env-secrets block, ECS task-definition secrets list, or EKS `valueFrom.secretKeyRef`). 6. Commit `cosign.pub` to `kaireonai-docs/security/cosign.pub` so downstream verifiers can validate signatures offline. ### Path B — Self-host / local (key file on disk) Use this for VM, on-prem, Docker Compose, or single-host installs. 1. `cosign generate-key-pair` (one time) → `cosign.key` + `cosign.pub`. 2. Mount `cosign.key` into the container (read-only, `0400`): ```yaml theme={null} # docker-compose.yml services: api: environment: COSIGN_KEY: "${COSIGN_KEY_BYTES}" # contents of cosign.key COSIGN_PASSWORD: "${COSIGN_PASSWORD}" # passphrase from step 1 ``` Or read at boot: ```bash theme={null} export COSIGN_KEY="$(cat /path/to/cosign.key)" export COSIGN_PASSWORD='' ``` 3. Restart the container. 4. Distribute `cosign.pub` to every party that needs to verify bundles (CI, audit reviewers, downstream consumers). > **Required.** Production deployments **must** install cosign signing > via Path A or Path B. Running without it leaves every provenance > bundle marked `X-Provenance-Signature: unsigned`, which downstream > verifiers reject. The fail-soft is intentional (a missing key never > breaks the response) but is not a substitute for installation. ## Verifying signing is live 1. Hit `/api/v1/decisions/:id/provenance` and confirm the response includes `X-Provenance-Signature: ` (not `unsigned`). 2. Verify offline: ```bash theme={null} curl -fsS .../api/v1/decisions/$ID/provenance > bundle.json curl -fsS .../api/v1/decisions/$ID/provenance -i | grep X-Provenance-Signature \ | sed 's/^[^:]*: //' | tr -d '\r' > sig.b64 cosign verify-blob --key cosign.pub --signature sig.b64 bundle.json ``` 3. Audit-log spot-check: `auditLog.changes.signed` should be `true` for every successful bundle. A `signatureFailureCode` field means the wire intended to sign but couldn't — page whoever owns the image / key-management layer. ## Roadmap * Keyless signing (Fulcio + OIDC + Rekor upload) so the deployment does not need a long-lived private key. * Image-time `cosign verify-blob` of the cosign release artefact itself (closes the bootstrap-trust gap noted above). * Reproducible-build verification step that re-canonicalises the bundle and re-signs against a CI-only key, comparing digests. See also: [Decision provenance bundle](../decisioning/decision-provenance) · [Compliance](./compliance)