Operate runtime feature flags (operator runbooks)

These runbooks cover the operator side of runtime feature flags: binding an environment, importing the legacy policy lineage, enrolling API and Project Management (PM), running the workload collector and the deployment coordinator, setting the expected inventory, changing, stopping and restoring a flag, recovery, and the return to Legacy. They are slice 6C.1 of the remaining delivery plan.

The feature behaviour, endpoint contracts and design history are in Managing Feature Flags. This guide does not repeat them. Where the two seem to differ, the code on main decides, and the difference is listed in Known gaps.

Related plans:

• Staging rehearsal plan and production prerequisites (6C.3, 6C.4);
• Isolated rehearsal checklist (6C.2, run in the local stack on 2026-10-22).

Status and boundary

NOT EXECUTED in staging or production. None of these runbooks has been run against staging or production. Nothing in this repository binds, imports, enrols or activates an environment. A merged PR is not evidence that an environment is ready. The isolated rehearsal (6C.2) ran them in a local E2E stack on 2026-10-22; its outcomes are in the isolated rehearsal checklist.

• Production candidate (decision D2). disableMembership is the first and only production candidate. It is the only key whose consumer contract is productionEligible: true, grain: Global, safeValue: false (src/charts/syrf-common/env-mapping.yaml; RuntimeProductionPolicy.Eligible). Statistics keys stay non-production.
• Deploy API and PM together (decision D4). See Mixed-version window.
• Rehearsals (decision D1). A rehearsal beyond the local stack may use a throwaway PR preview environment. Staging and production remain written plans until Chris approves each step.
• Production prerequisite. Production enables and restores need RuntimeFeatureFlags:FleetInventoryMode=Collector and a running collector (runbook 4). Since #4407 (server-time retirement, merged as 9174f9e9), production admission, the fleet view and deployment completion follow the configured inventory mode. The default is still Static, which keeps every old pod's acknowledgement row: after any Kubernetes rollout those rows stop the fleet reaching Active, so production enables and restores are refused in Static mode. The remaining prerequisite is operational: set Collector mode and keep the collector publishing.
• Forward only. Deployments and data only move forward (ADR-023). A rollback is a planned replacement release. Every "roll-forward" column in the plans means a new reviewed change, never an undo.
• GitOps only. Configuration changes are cluster-gitops PRs. Never use kubectl apply, helm install or direct database edits.

Before you start

Placeholders

Placeholder	Meaning
<env>	Binding environment ID: production, staging, preview:pr-<n> or local:<stack>
<runtime-env>	The RuntimeEnvironment the binding runs as: the part of <env> before the colon
<database>	The configured MongoDB database name
<cluster-fingerprint>	The live cluster fingerprint that status prints, for example <setName>@<hostSuffix>
<binding-epoch>	The binding's epoch GUID, from the bind log line or the authority identity
<api-origin>	The API's HTTPS origin, with no path
<identity-origin>	The Identity service origin (OpenIddict environments only)
<admin-session>	An authenticated real administrator: browser session or user bearer token. Never a machine token
<report-token>, <collector-token>, <coordinator-token>	Machine access tokens, one scope each (runbooks 4 and 5)
<key>	The flag key. For production this is always disableMembership
<identity>	The complete policy identity object, copied unchanged from authority-status (below)
<mutation-id>	A new GUID per intended change. Reuse it only to retry the same request

Never put real values, tokens or secrets in this document, a PR description or a log excerpt.

How settings reach a host

The .NET hosts read SYRF__-prefixed environment variables, with __ meaning :. So the setting RuntimeFeatureFlags:Mode is the variable SYRF__RuntimeFeatureFlags__Mode.

Only two groups of these settings are chart values:

• migrations.enabled and migrations.binding.* (API and PM charts), rendered into the migration Jobs as SYRF_MIGRATION_BINDING_EXPECTED_*;
• runtimePolicyHostMetadata.enabled (API and PM charts), which renders InstanceUid, ImageVersion, ExpectedReplicas and Autoscaled from the Deployment.

Every other RuntimeFeatureFlags:* setting in these runbooks has no env-mapping.yaml entry. It reaches the host only as a raw env: entry in the service's cluster-gitops values. Raw env: entries are literal strings (_deployment-dotnet.tpl), so they must never carry a secret.

TO CONFIRM: whether reviewed chart values should replace these raw env: entries before production.

The policy identity

Every policy request carries the complete current identity. Read it from the API, then copy it unchanged:

GET /api/runtime-feature-flags/authority-status
<admin-session>

Expected (Shared mode, healthy):

{
  "mode": "Shared",
  "host": "api",
  "fleetScope": "local-process",
  "state": "Ready",
  "identity": {
    "environmentId": "<env>",
    "bindingEpoch": "<binding-epoch>",
    "storeEpoch": "<store-epoch>",
    "revision": "<revision>",
    "catalogDigest": "<catalog-digest>",
    "baselineDigest": "<baseline-digest>",
    "policyDigest": "<policy-digest>"
  },
  "observedAtUtc": "<time>",
  "lastAttemptAtUtc": "<time>",
  "reason": null
}

revision is a decimal string. state is one of Ready, Uninitialized, Stale (MaximumAgeExceeded), Unavailable (AuthorityNotInitialized, AuthorityReadFailed), Legacy, Quarantined (InvalidBinding, AuthorityNotInitialized, IdentityOrRevisionRegression) or, in Legacy mode, Disabled with reason LegacyMode, or LegacyModeAuthorityPreserved once that Legacy process has found an initialized authority (runbook 10). This is one API process's view, not the fleet's.

Retry rules

• A 503 with runtime_policy_commit_unknown, a lost response or a timeout does not prove failure. Retry the same mutation ID and the unchanged request, or look the outcome up (R7.7).
• Never mint a new mutation ID because an outcome is unknown.
• A 409 runtime_policy_revision_conflict means the identity moved. Read it again and decide again.

Exit codes of the operator commands

Command	0	1	2	3
Migration entry point (SYRF_MIGRATION_MODE)	Success	Failure (verify problems)	Usage or binding refusal; nothing written	Lease or writer wait timed out
Lineage command (SYRF_RUNTIME_POLICY_MODE)	Artifact written	Unexpected failure; retry the identical reviewed plan	Refused (Runtime policy lineage refused: …)	n/a
Historical import (SYRF_RUNTIME_RELEASE_HISTORY_MODE)	Artifact written	Unexpected failure; retry the identical reviewed preview	Refused (Historical release import refused: …)	n/a
scripts/capture-runtime-workload.py	Capture written	n/a	Capture refused: …	n/a
scripts/publish-runtime-workload.py	Receipt matched	HTTP 3xx/4xx refusal (not 408/425/429)	Local refusal	Outcome unknown; replay the exact capture
scripts/run-runtime-workload-collector.py (R4.7)	Stopped cleanly (SIGTERM)	n/a	Collector refused: … at start-up; nothing sent	n/a
scripts/runtime-compatibility-check.py	pass	refuse	insufficient-evidence	unusable-input

Mixed-version window (D4)

When application authority is enforced, an API that asks an older PM about the operation-specific runtime capabilities gets a retryable 503 on the flag-admin endpoints until both run the same version (#4403, from #4354). The 503 is ProblemDetails with code: "authority-unavailable" and a Retry-After header. So:

• promote and sync API and PM at the same build, in one change;
• do not start any runbook step that calls an admin route until both services report the same image;
• treat a 503 authority-unavailable during a rollout as the expected window, not a failed step.

ADR-023 also expects API and PM to run different builds briefly during a rollout. Migrations stay additive for that window.

Evidence store

Keep every artifact named in "Evidence to capture" with the run record (see the approval record template). Preview and receipt files are review evidence: never edit or reformat them.

Runbook order

#	Runbook	Writes to the environment
1	Environment binding	Binding record (once)
2	Lineage preview and apply with writer freeze	Authority document and receipt
3	Consumer enrolment	Deployment configuration
4	Collector identity and scope	Workload observations
5	Coordinator identity	Deployment constraints
6	Expected inventory and Collector mode	Deployment configuration
7	Change one flag	Policy revision and receipt
8	Emergency stop and restore	Policy revision and receipt
9	Recovery and rebind	Depends on the case
10	Legacy return	Deployment configuration
11	Historical release import (written only)	One release record and one event

Runbooks 1 to 3 run once per environment, in that order. Runbook 2 needs a full API outage (no API process may be registered), and runbook 3 must follow it before the outage ends: an API in Legacy mode starts against an imported authority but serves only the deployed values (runbook 10), so the saved runtime policy applies only once runbook 3 has enrolled the hosts.

Runbook 4 is numbered after runbook 3 but must be working before runbook 2 starts. Runbook 3 sets Collector mode, and a Collector-mode fleet is Active only while both services have observations younger than 30 seconds. So configure the collector (R4.1–R4.2) and prove one capture and publication per service (R4.3–R4.5) first, then keep publishing from the moment the API is back (R3.4). The continuous collector (R4.7) does that unattended. While the API is at zero it backs off (at most 30 seconds between attempts) and resumes when the API returns.

1. Environment binding

Purpose

Bind the database once to one environment ID, database name and cluster fingerprint. Migrations 0009 to 0021 declare RequiresBinding, and every runtime authority read and write checks the binding.

Preconditions

• The service images contain migrations 0009–0021 (check migrations.lock.json at the image's commit).
• The ADR-023 pre-deploy migrations are available in the API and PM charts (migrations.enabled), and the promoted images contain #4740's change: the chart renders migrate-bind and migrate-up init containers on each Deployment, besides the PreSync Jobs. Check that the rendered Deployment has initContainers (R1.3). Without them, a values-only change can land with no migration run at all (#4740).
• For staging: Chris has approved the step (owner gate, plan §8). For production: Chris approves the specific cluster-gitops PR.
• No binding exists yet, or the existing one is identical. bind never overwrites a different binding.
• PR previews: previews do not run up or bind today. Their database lifecycle Job runs index-init only, and the migration Job refuses preview.waitForDatabase. TO CONFIRM: how a D1 preview rehearsal gets bind and up.

Steps

1. R1.1 Read the live fingerprint. Run the API or PM image once with:

SYRF_MIGRATION_MODE=status
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>        # optional on this first run
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>      # optional
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>  # optional

TO CONFIRM: no chart template renders a status Job. In the local stack e2e/scripts/pm-migrations.sh runs it. For staging and production, the reviewed way to run a one-off status (and verify) is not defined. 2. R1.2 Prepare the cluster-gitops change for both the API and the PM service values:

migrations:
  enabled: true
  binding:
    bind: true
    expectedEnvironment: "<env>"
    expectedDatabase: "<database>"
    expectedCluster: "<cluster-fingerprint>"

Copy expectedCluster from R1.1. Both charts need the three expected values, because each chart's up Job and migrate-up init container read them when a RequiresBinding migration is pending. bind: true adds a PreSync bind Job in sync wave −1, ahead of up, and a migrate-bind init container ahead of migrate-up. A second identical bind is a no-op. Whether bind stays set after the first bind is open (#4263).

While bind: true stays set, every pod start runs migrate-bind. That means the environment and database checks and a live hello fingerprint, even when the binding already exists. A hello failure delays every restart and scale-up until Mongo answers. A fingerprint, environment or database mismatch stops every new pod with exit 2; for PM (Recreate, one replica) that is an outage until a fix lands. Recommended: once the first bind has succeeded and R1.5 verify passes, set migrations.binding.bind: false in a follow-up cluster-gitops change. Keep the three expected values: up still checks the binding before any RequiresBinding migration. The change rolls the pods once (ADR-023 amendment 2026-11-04, Consequences). 3. R1.3 Review the render in the cluster-gitops PR diff: one bind Job and one up Job per chart, and migrate-bind then migrate-up under initContainers on each Deployment, all with the three SYRF_MIGRATION_BINDING_EXPECTED_* variables. 4. R1.4 Merge with the required approval and let ArgoCD sync. Do not sync API and PM at different builds (D4). Nothing else is needed, and the result does not depend on how Argo runs the sync: - A full sync runs the PreSync bind and up Jobs, then the Deployment. The init containers then find everything in place and finish in about a second. - A self-heal sync of only the Deployment skips every hook. Argo CD v3.2 does this for a change that lives only in values files, which this one is (ADR-023 amendment 2026-11-04). The new pod's init containers bind and migrate before the app container starts.

Do not request a sync, patch the Application or create a Job by hand to make the hooks run. The 3 November recovery did that, and it broke the GitOps-only rule (#4740). Recovery is always a cluster-gitops commit (see If a step is refused). 5. R1.5 Verify with SYRF_MIGRATION_MODE=verify (same TO CONFIRM as R1.1), or, once reports are enabled (runbook 4), with the report's migrations prerequisite.

Preflight checks and expected output

Check	Expected output
R1.1 status, no binding yet	Environment binding: none in database <database>; live cluster fingerprint <cluster-fingerprint> and, without expected values, Environment binding: no SYRF_MIGRATION_BINDING_EXPECTED_* value is set; nothing compared. Exit 0
R1.1 status with expected values set	Environment binding matches the expected environment '<env>', database '<database>', cluster '<cluster-fingerprint>', or one Environment binding mismatch: … warning per problem. Exit 0 either way
R1.4 bind Job log, or the migrate-bind init container (kubectl logs <pod> -c migrate-bind)	Environment bound: <env>, database <database>, cluster <cluster-fingerprint>, epoch <binding-epoch>, announcement delivery <mode>, where <mode> is production in every environment since D60 A (26 Oct 2026; a binding written earlier keeps testSink or disabled, which works unchanged and is never rewritten). On a repeat: Environment binding already in place: …; nothing changed. Exit 0
R1.4 up Job log, or migrate-up (kubectl logs <pod> -c migrate-up)	Whichever runner takes the lease first: Environment binding verified: <env>, database <database>, cluster <cluster-fingerprint>, epoch <binding-epoch>, then each migration applied, then Migrations up to date: schema version <n>, <k> changed. A runner that waited for the lease logs Migration lease project-management is held by another runner; waiting up to …, then … 0 changed. A later pod start logs Migrations up to date: schema version <n>, 0 changed (nothing pending, so the lease was not taken). Exit 0
R1.5 verify	Verify: every migration this binary knows is applied and every declared index exists. Exit 0

Evidence to capture

• The status output from R1.1 (fingerprint).
• The cluster-gitops PR URL and its approval.
• The bind and up logs, including the binding epoch: from the Jobs if they ran, otherwise from the first new pod's migrate-bind and migrate-up init containers.
• The verify output, or a report whose migrations prerequisite is Ready.

If a step is refused

Every binding refusal exits 2 and writes nothing. For up there is no partial apply, and no pod serves an unmigrated schema. What keeps serving depends on how the change arrived:

• PreSync Job failed (full sync). The sync stops before the Deployment changes, so the previous release keeps serving.
• Init container failed (any sync of the Deployment). The new pod stays in Init:Error or Init:CrashLoopBackOff and its app container never starts. The kubelet retries with back-off. Under RollingUpdate (the API) the old pods keep serving. Under Recreate with one replica (PM) the old pod has already gone, so PM is unavailable until the fix lands.

Recovery is a cluster-gitops commit, never a manual Job. Read the message (kubectl logs <pod> -c migrate-bind or -c migrate-up), fix the values or revert the change, and merge with the required approval. Argo applies the corrected Deployment, and its init containers run again. If the fault cannot be fixed in values, migrations.enabled: false is the kill switch: the app then builds its own indexes at startup. Never kubectl create a Job or patch the Application to force hooks (#4740).

Message (prefix Bind refused: or Up refused before applying anything (…):)	Action
SYRF_MIGRATION_BINDING_EXPECTED_… is not set; … are all required.	Set all three values
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT must be production, staging, preview:pr-<number> or local:<stack> (got '…')	Fix the environment ID
the configured RuntimeEnvironment '…' does not match the expected environment '…'	The values file renders the wrong environment. Stop and investigate; never change the expected value to fit
the configured database '…' does not equal the expected database '…'	Same as above
the live cluster fingerprint '…' does not equal the expected cluster '…'	Wrong cluster, or the cluster changed. See runbook 9
database '…' is already bound to …; a different binding is never overwritten.	Runbook 9. Do not edit the record
database '…' has no environment binding; run SYRF_MIGRATION_MODE=bind first.	Add the bind Job (R1.2)
SYRF_MIGRATION_MODE must be one of status, up, down, verify, bind (got '…')	Fix the mode
Exit 3	The migration lease or writer wait timed out. Find the other runner (the lease owner names its pod) before retrying. An init container that exits 3 is restarted by the kubelet on its own

2. Lineage preview and apply with writer freeze

Purpose

Turn the legacy runtimeFeatureFlagOverrides document into a bound runtime authority with a reviewed baseline, keeping the legacy revision and overrides exactly. This is an operator command in the API image, separate from migrations and start-up. It does not enrol any consumer.

Two branches. Where the legacy document exists, import it (R2.1–R2.7 below). Where no global flag document exists at all, as in production, use Initialise (no legacy document) instead: it creates the authority from the deployed values (owner decision D8=A). Check which applies with the first preflight of each branch. The two never mix: import refuses a missing document and initialise refuses any document.

Preconditions

• Runbook 1 is complete and the binding matches.
• Migrations 0009 and 0010 are applied with their expected definitions. The command never creates indexes.
• The legacy policy document exists (_id: "global"). Where there is none (production: legacy mutation is disabled there), do not create one: use Initialise (no legacy document).
• syrf_metadata has a well-formed syrf-api instance registry.
• The image used for the command is the API build that will serve the imported authority. The catalog digest is part of the identity; a preview made with another catalog is rejected later.
• The consumer enrolment change (runbook 3) is reviewed and ready to merge straight after apply.
• Approval: Chris for staging and production. D1 preview rehearsals follow the staging plan.
• The promoted API chart has the runtimePolicyLineage Job (#4794). In staging and production the command runs only through it, by cluster-gitops values PRs (see Delivery through GitOps). Never run it from a Job, pod or kubectl run made by hand. The Job runs the import branch only. Initialise has no GitOps delivery yet: it needs the API's rendered FeatureFlags settings, which the Job does not carry.

Delivery through GitOps

The API chart's runtimePolicyLineage values render a ConfigMap holding the reviewed input byte for byte and one ordinary Job (not an Argo CD hook). The Job runs the API image's entry point with the R2.2 or R2.5 variables and prints the artifact to its log between two marker lines. Its name is syrf-api-runtime-policy-lineage-<mode>-<hash>, where <hash> is the first 10 hex digits of the input's SHA-256. So a new input makes a new Job, and an unchanged input keeps the existing Job, which never runs again. The paths inside the Job are fixed: the input is /reviewed/input.json (the ConfigMap, read-only) and the output is /artifact/lineage-preview.json or /artifact/lineage-receipt.json (an emptyDir, new for every pod). The chart reference is in src/services/api/.chart/README.md.

G1 Preview PR (R2.2). One cluster-gitops PR on the API service values for the environment:

migrations:
  binding:                      # already set by runbook 1; the render fails if any is empty
    expectedEnvironment: "<env>"
    expectedDatabase: "<database>"
    expectedCluster: "<cluster-fingerprint>"
runtimePolicyLineage:
  enabled: true
  mode: preview
  input: |-
    {
      "values": { "<key-1>": false, "<key-2>": true },
      "approval": "<reviewed baseline decision reference>",
      "provenance": "<source configuration and report identities used to reconcile it>"
    }

Review the rendered diff: one ConfigMap and one Job, both named …-preview-<hash>; the Job has the R2.2 variables and no SYRF_MIGRATION_MODE; the image is the build that will serve the authority. Merge with the required approval. The API keeps serving: preview is read-only.

G2 Read the preview log. Read-only, before anything else changes, because the log goes when the Job does:

kubectl logs job/syrf-api-runtime-policy-lineage-preview-<hash> -n <namespace> > lineage-preview.log

Exit 0 shows the R2.2 log line, then ----- BEGIN SYRF RUNTIME POLICY ARTIFACT /artifact/lineage-preview.json -----, the artifact, and ----- END SYRF RUNTIME POLICY ARTIFACT /artifact/lineage-preview.json sha256=<sha256> bytes=<n> -----. Keep the log as evidence. Cut the artifact out with its exact size from the END line, then check it:

sed -n '/^----- BEGIN SYRF RUNTIME POLICY ARTIFACT /,/^----- END SYRF RUNTIME POLICY ARTIFACT /p' lineage-preview.log \
  | sed '1d;$d' | head -c <n> > lineage-preview.json
sha256sum lineage-preview.json      # must print the END line's <sha256>

The artifact has LF line endings only, so this copy is the file byte for byte. The container log path does not keep CR bytes, and an API image built before #4820 wrote the artifact with CRLF (the MongoDB driver's default). For a preview from such an image, the copy above is short and its SHA-256 differs. Put the CR back before each LF instead, then check it as above:

sed -n '/^----- BEGIN SYRF RUNTIME POLICY ARTIFACT /,/^----- END SYRF RUNTIME POLICY ARTIFACT /p' lineage-preview.log \
  | sed '1d;$d' | awk '{printf "%s\r\n", $0}' | head -c <n> > lineage-preview.json

The line endings change no digest: previewDigest and the receipt digests are over the document's BSON, not the file's bytes, so apply accepts either form of the same preview.

Review the file as R2.3 describes. A refusal prints ----- NO SYRF RUNTIME POLICY ARTIFACT: … (command exit 2) ----- after the Runtime policy lineage refused: … line; the Job fails, and nothing was written.

G3 Freeze (R2.4) through its own cluster-gitops PR, and wait for zero API pods. Do not combine it with G4: an apply that starts before the last API pod has gone is refused, and that Job does not run again (see G6).

G4 Apply PR (R2.5). Replace the preview values with the exact preview file:

runtimePolicyLineage:
  enabled: true
  mode: apply
  actor: "<operator identity>"
  writerFreezeAcknowledgement: "<freeze reference from the approval record>"
  input: |-
    <lineage-preview.json, unchanged, each line indented by four spaces>

|- keeps the file's bytes: the artifact has no final newline. The render fails if the actor or the acknowledgement is empty. In the rendered diff, the annotation syrf.org.uk/runtime-policy-input-sha256 on the new …-apply-<hash> ConfigMap and Job must equal the <sha256> from G2. If it differs, the input is not the reviewed preview: stop and fix the PR. Merge with the required approval. Argo CD prunes the preview Job and ConfigMap.

YAML turns every line break in the |- block into LF. For a preview from an image built before #4820 (CRLF, see G2) the annotation is therefore the SHA-256 of the LF copy, not the END line's <sha256>: check it against tr -d '\r' < lineage-preview.json | sha256sum instead. Apply reads the same document either way.

G5 Read the receipt. As G2, with job/syrf-api-runtime-policy-lineage-apply-<hash>: the R2.5 log line, then the receipt between the markers (for an image built before #4820, use G2's CR-restoring extraction for the receipt too). Save the log and the cut-out lineage-receipt.json with the evidence, and check them as the R2.5 receipt row describes.

G6 Switch it off. Go to runbook 3 (R2.7). In the same change, or straight after it, set runtimePolicyLineage.enabled: false (or remove the block). Argo CD prunes the Job and the ConfigMap. Kubernetes also deletes a finished Job 24 hours after it finished (ttlSecondsAfterFinished). In an environment with automated self-heal (staging) Argo CD then creates the Job again, and it runs again with the same input. That is harmless: a preview is read-only, a committed import returns its original receipt without writing (R2.6), and every refusal writes nothing. It is noise in the evidence, so switch the Job off within the day.

Retry (R2.6). An unknown apply outcome (exit 1, or a log lost before it was read) is retried with the identical preview. An unchanged input keeps the same Job, which never runs again. So save whatever log exists, merge enabled: false, wait for Argo CD to prune the Job, then merge the identical apply values again. The new pod has a fresh emptyDir, so the output path is new, as R2.6 requires.

Rollback. A preview writes nothing: switch the Job off. A refused apply (exit 2) writes nothing: fix the cause (table below), make and review a new preview if the message asks for one, and continue from G1. A committed import is not undone by reverting the values PR. Reverting only removes the Job. Deployments and data move forward only. If the imported authority must not be used, keep the hosts in Legacy mode (runbook 10) and decide the next step with the approver.

Never:

• edit, reformat or re-indent the preview between G2 and G4, or type it in by hand. Only the G4 |- indentation is allowed, and the SHA-256 check proves it changed nothing;
• re-run apply with a different input: a new preview always means a new review from G1;
• create, patch, delete or re-run the Job by hand (kubectl create job --from, kubectl delete job), or request an Argo CD sync to force it. Every change is a values PR;
• change the API image (a promotion) while the Job is enabled. The Job's pod template cannot change, so Argo CD fails to apply it. A preview made with another image's catalog is rejected later anyway;
• set SYRF_RUNTIME_POLICY_* or SYRF_MIGRATION_* through env:. The chart ignores those names for this Job.

Steps

1. R2.1 Write the reviewed baseline JSON. It needs a Boolean for every key of the generated catalog (54 keys on main), plus approval and provenance text:

{
  "values": { "<key-1>": false, "<key-2>": true },
  "approval": "<reviewed baseline decision reference>",
  "provenance": "<source configuration and report identities used to reconcile it>"
}

List the keys with:

grep -oE '^        new\("[A-Za-z0-9]+", "' \
  src/libs/mongo/SyRF.Mongo.Common/RuntimeFeatureFlags/RuntimeFeatureFlagCatalog.generated.cs \
  | sed -E 's/^ +new\("([A-Za-z0-9]+)".*/\1/'

The keys are the first argument of each new("<key>", …) entry in that generated file, which is the catalog's key list; read it there if grep is unavailable.

Reconcile each value against the rendered values of every consuming service. The command never takes the API's current settings as the baseline. 2. R2.2 Preview (read-only). Through G1 and G2. The Job runs the API image's normal entry point once with the usual Mongo configuration and RuntimeEnvironment=<runtime-env>, plus:

SYRF_RUNTIME_POLICY_MODE=preview
SYRF_RUNTIME_POLICY_INPUT=/reviewed/input.json            # the baseline JSON (ConfigMap)
SYRF_RUNTIME_POLICY_OUTPUT=/artifact/lineage-preview.json
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>

SYRF_MIGRATION_MODE must not be set at the same time. 3. R2.3 Review the preview file. Check kind: legacyLineageImport, planVersion: 1, sourceRevision (equal to the legacy revision), target.environmentId = <env>, target.bindingEpoch = <binding-epoch>, a new target.storeEpoch, the baseline values and previewDigest. It is Canonical Extended JSON: do not reformat it. Numbers carry their type wrapper, so planVersion reads {"$numberInt": "1"} and target.revision {"$numberLong": "<revision>"}; sourceRevision is a decimal string; the baseline values are in target.baseline. If the legacy document saves an override for a key that was settled or retired from the catalog (RuntimeFeatureFlagKeys.RemovedFlags), the preview has a top-level droppedRemovedOverrides listing each such key with its legacy value, and target.overrides leaves them out: a removed key is never evaluated, so the import does not carry it into the authority. Check that every listed key is one you expect to have been settled or retired; the removedFlags comment in src/charts/syrf-common/env-mapping.yaml says where each key came from (some were retired by hand before that list existed, such as apmEnabled). With nothing to drop the field is absent. sourcePolicyDigest and sourceDocumentDigest still cover the full legacy document, and apply refuses if this image would drop a different set; the receipt records the same droppedRemovedOverrides beside sourceOverrides (the imported overrides). Any other key outside the catalog refuses the preview. 4. R2.4 Freeze writers. Start the freeze recorded in the approval: no cluster-gitops merges for this environment except this procedure, no auto-promotion, no autoscaling. Scale the API to zero replicas through cluster-gitops and wait until every API pod has terminated. TO CONFIRM: the freeze mechanism for staging's auto-merged promotion PRs. 5. R2.5 Apply the exact reviewed preview with the same image and binding values, through G4 and G5:

SYRF_RUNTIME_POLICY_MODE=apply
SYRF_RUNTIME_POLICY_INPUT=/reviewed/input.json            # the exact preview (ConfigMap)
SYRF_RUNTIME_POLICY_OUTPUT=/artifact/lineage-receipt.json
SYRF_RUNTIME_POLICY_ACTOR=<operator identity>
SYRF_RUNTIME_POLICY_WRITER_FREEZE_ACK=<freeze reference from the approval record>
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>

1. R2.6 If the outcome is unknown (exit 1, lost output), rerun R2.5 with the identical preview and a new output path (Retry). A committed import returns its original receipt without writing again.
2. R2.7 Go straight to runbook 3, then end the freeze. An API restarted in Legacy mode would start, but it serves only the deployed values and refuses legacy changes (runbook 10); the imported policy applies only in Shared mode. An older image (before #4414) instead refuses to start against the imported authority, and it writes its syrf-api writer-registry entry (DatabaseMetadataWriter) before it refuses, so a refused start still leaves an entry behind (seen in the 6C.2 rehearsal).

Preflight checks and expected output

Check	Expected output
Before R2.2: legacy document	db.runtimeFeatureFlagOverrides.findOne({_id: "global"}, {_id: 1, revision: 1}) returns a document with no storeEpoch field (read-only query)
Before R2.5: writer registry	db.syrf_metadata.findOne({_id: "meta"}, {"services.syrf-api.instances": 1}) shows instances: {}. Apply checks this itself, twice
Before R2.5: no API pods	The API Deployment reports 0 ready and 0 total pods
R2.2 log	Runtime policy lineage preview completed; artifact: /artifact/lineage-preview.json. No consumer was enrolled. Exit 0, then the artifact between the BEGIN and END lines (G2)
R2.5 log	Runtime policy lineage apply completed; artifact: /artifact/lineage-receipt.json. No consumer was enrolled. Exit 0, then the receipt between the BEGIN and END lines (G5)
R2.5 receipt	kind: legacyLineageImport, the preview's mutationId and previewDigest, environmentId, bindingEpoch, storeEpoch, revision equal to the source revision, actor, writerFreezeAcknowledgement, recordedAtUtc

A registry entry from a pod that did not shut down cleanly stays until a later API start prunes entries older than 24 hours (DatabaseMetadataWriter). There is no bypass. The one entry the freeze does not count is a Legacy API that has already found an initialized authority: it marks its own registration runtimePolicyReadOnly (bound to that registration's connectedAt) because it can never write runtime policy again (runbook 10). Every other entry, including one from an older image, is counted. TO CONFIRM: the approved way to clear a stale entry younger than 24 hours, other than waiting.

Evidence to capture

• The baseline JSON, its approval and provenance references.
• The cluster-gitops PR URLs for G1, G3, G4 and G6, and both Job logs (G2, G5).
• The preview file, its previewDigest, and its SHA-256 from the END line (equal to the G4 annotation; for a CRLF preview from an image built before #4820, the G4 annotation is its LF copy's SHA-256, see G4).
• The freeze reference, start and end times, and proof of zero API pods.
• The receipt file and the command logs (no secrets).

If a step is refused

Refusals exit 2 with Runtime policy lineage refused: <message>. Nothing is written.

Message	Action
SYRF_RUNTIME_POLICY_MODE must be preview or apply. / … is required.	Fix the invocation
Lineage import and schema migration modes cannot be combined.	Unset SYRF_MIGRATION_MODE
Input and output paths must differ; preserve the reviewed input. / Output already exists; choose a new receipt/preview path.	Use a new output path
Input must be an existing JSON file of at most 1 MiB.	Fix the input path or file
Lineage import requires an exact verified environment/database/cluster binding: …	Runbook 1, or runbook 9
Missing legacy policy; lineage import cannot invent an initial state.	Initialise the legacy state with a reviewed edit first
Legacy source has unknown fields; review its schema before importing.	Stop; an unknown legacy shape needs review (it may already be imported)
API writer registry is missing or malformed; absence of evidence is not a writer freeze.	Investigate the registry; do not create it by hand without approval
API writers remain registered. Stop/freeze all legacy writers before lineage import; there is no bypass switch.	Finish R2.4
Required migration-owned audit index is missing or incompatible: …	Apply 0009/0010 (runbook 1)
The reviewed lineage preview was modified or has an unsupported version.	Make and review a new preview
The reviewed lineage preview belongs to a different environment binding. / Environment binding changed after preview.	Make a new preview against the current binding
Legacy source changed after preview; obtain and review a new preview.	A writer changed the policy: the freeze failed. New preview
Lineage import must preserve the exact legacy revision and overrides. / Legacy source audit references changed after preview.	New preview
The preview's droppedRemovedOverrides differs from the stored overrides for keys this binary has removed; obtain and review a new preview with this image.	The preview was edited, or made by an image with a different removed-flag list. New preview with the apply image
Policy values must use known catalog keys; the reviewed baseline must contain every key. (preview)	The baseline misses or adds a key, or the legacy document saves an override for a key that is neither catalogued nor removed. Review the document; do not edit it without approval
Mutation ID was already used with a different lineage plan.	Never reuse a preview's mutation ID for a different plan
Too many ambiguous source audit references; review the legacy history before importing.	Stop and review the audit history

Initialise (no legacy document)

Purpose. Create the first bound runtime authority where no global flag document exists. The baseline is the deployed values: the FeatureFlags configuration the API image renders for this command, for every catalog key. The authority starts at revision 0 with no overrides and no stops. Production uses this branch instead of the import (D8=A): legacy mutation is disabled there, so it has nothing to import. The command, its digest discipline, writer freeze, receipts and output path rules are the import's; only the source differs.

Preconditions. As for the import, except:

• No global document exists, legacy or initialized. The command refuses either.
• Every migration the image knows is applied and every declared index exists (verify is clean; runbook 1). The command runs the same read-only check and refuses otherwise. It never creates indexes.
• The command runs with the rendered values the API will serve: the same image and the same FeatureFlags settings as the enrolled API (runbook 3). Reconcile them with Project Management's rendered values before review. The command reads only the API's.
• The same image and settings run preview and apply. Apply reads the deployed values again and refuses if they changed.

Steps.

1. R2.I1 Write the reviewed input. Only approval and provenance; the baseline is never supplied by hand:

{
  "approval": "<owner approval reference for the initial baseline>",
  "provenance": "<rendered values (API and PM) reconciled for this baseline>"
}

1. R2.I2 Preview (read-only). As R2.2, plus SYRF_RUNTIME_POLICY_SOURCE=initialize:

SYRF_RUNTIME_POLICY_MODE=preview
SYRF_RUNTIME_POLICY_SOURCE=initialize
SYRF_RUNTIME_POLICY_INPUT=/reviewed/initialise-input.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/initialise-preview.json
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>

The preview is deterministic: the same input, image, values, binding and migrations always give a byte-identical file (the store epoch and mutation ID are derived from them). Run it twice to two paths and compare (cmp) to prove nothing varies. 3. R2.I3 Review the preview file. Check kind: deployedValueInitialization, planVersion: 1, source: deployedValues, migrationCatalogueDigest, target.revision: 0, target.overrides: {}, target.killSwitches: {}, target.environmentId = <env>, target.bindingEpoch = <binding-epoch>, target.storeEpoch, previewDigest, and every value in target.baseline against the reconciled rendered values. Do not reformat it. 4. R2.I4 Freeze writers. As R2.4. 5. R2.I5 Apply the exact reviewed preview with the same image, values and binding:

SYRF_RUNTIME_POLICY_MODE=apply
SYRF_RUNTIME_POLICY_SOURCE=initialize
SYRF_RUNTIME_POLICY_INPUT=/reviewed/initialise-preview.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/initialise-receipt.json
SYRF_RUNTIME_POLICY_ACTOR=<operator identity>
SYRF_RUNTIME_POLICY_WRITER_FREEZE_ACK=<freeze reference from the approval record>
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>

1. R2.I6 If the outcome is unknown, rerun R2.I5 with the identical preview and a new output path. A committed initialise returns its original receipt without writing, even after the API is back. Any other preview is refused once the authority exists.
2. R2.I7 As R2.7: go straight to runbook 3, then end the freeze.

Preflight checks and expected output.

Check	Expected output
Before R2.I2: no global document	db.runtimeFeatureFlagOverrides.countDocuments({_id: "global"}) returns 0 (read-only query)
Before R2.I2: migrations	SYRF_MIGRATION_MODE=verify prints Verify: every migration this binary knows is applied and every declared index exists, exit 0
Before R2.I5: writer registry, no API pods	As for the import (R2.5 rows above)
R2.I2 log	Runtime policy initialise preview completed; artifact: /reviewed/initialise-preview.json. No consumer was enrolled. Exit 0
R2.I2 repeated	A second preview to another path is byte-identical (cmp prints nothing, exit 0)
R2.I5 log	Runtime policy initialise apply completed; artifact: /reviewed/initialise-receipt.json. No consumer was enrolled. Exit 0
R2.I5 receipt	kind: deployedValueInitialization, _id and mutationId equal to the preview's mutationId, the preview's previewDigest, source: deployedValues, migrationCatalogueDigest, sourceOverrides: {}, baseline, environmentId, bindingEpoch, storeEpoch, revision: 0, actor, writerFreezeAcknowledgement, recordedAtUtc
After R2.I5	A new preview is refused with An initialized runtime policy authority already exists; initialise never replaces it. (exit 2)
After runbook 3	authority-status is Shared/Ready at the receipt's store epoch and revision 0

Evidence to capture. The input file, both preview files and the cmp result, the reconciled rendered values, the freeze reference and zero-pod proof, the receipt, and the command logs (no secrets).

If a step is refused. Refusals exit 2 with Runtime policy lineage refused: <message>. Nothing is written. The import's invocation, path, binding and writer messages (table above) apply unchanged. Initialise adds:

Message	Action
SYRF_RUNTIME_POLICY_SOURCE must be import or initialize when set.	Fix the invocation
A legacy runtime policy document already exists; initialise never replaces it. Import it with the lineage import (SYRF_RUNTIME_POLICY_SOURCE unset) instead.	Use the import branch
An initialized runtime policy authority already exists; initialise never replaces it.	Stop: the environment is already initialised. If this was a retry, use the identical preview (R2.I6)
Migrations are not Ready; apply every migration this binary knows before initialising: …	Runbook 1 (up, then verify)
Initialise needs this binary's migration catalogue to prove that migrations are Ready.	Use the API image's entry point; the catalogue is built in
Initialise input takes only approval and provenance; the baseline is always the deployed values.	Remove values (and anything else) from the input
Deployed values changed after preview; obtain and review a new preview.	The apply ran with other FeatureFlags values than the preview: use the reviewed image and values, or make a new preview
The migration catalogue changed after preview; obtain and review a new preview.	The image changed between preview and apply: new preview
The reviewed plan was made for another lineage source; use the SYRF_RUNTIME_POLICY_SOURCE it was previewed with.	Set SYRF_RUNTIME_POLICY_SOURCE to match the preview
An initialise plan must create revision 0 from the deployed values, with no overrides and no stops.	The preview was altered: discard it, make a new one

3. Consumer enrolment

Purpose

Switch API and PM to the bound runtime authority (RuntimeFeatureFlags:Mode=Shared) with deployment-owned host metadata, so that their decisions and acknowledgements follow the imported policy.

Preconditions

• Runbook 2 is complete, and the freeze from R2.4 is still held with the API at zero.
• API and PM images are the same build (D4), and that build matches the catalog in the receipt.
• Migration 0011 is applied (acknowledgement index).
• Runbook 4 is ready: the API's collector settings (R4.1) are deployed, a token can be obtained (R4.2), and one capture and publication per service (R4.3–R4.5) has succeeded. R3.1 sets Collector mode, so the fleet can only be Active while both services have fresh observations (younger than 30 seconds).
• The continuous collector (R4.7) is running, or an operator is ready to run the manual capture-and-publish loop (R3.4) as soon as the API is back. While the API is at zero nothing can be published, because the observations endpoint is on the API. The collector retries with bounded backoff and resumes by itself.
• Approval for the cluster-gitops change.
• Staging only: staging values may set SYRF__RuntimeFeatureFlags__SharedReaderMode on API and PM (they did in cluster-gitops on 2026-10-17). Shared mode requires Disabled, so the same change must set it.

Steps

1. R3.1 Prepare one cluster-gitops change for API and PM. Shape for the API service values (PM is the same without the API-only lines):

replicaCount: <api-replicas>        # fixed, positive
hpa:
  enabled: false                    # an autoscaled fleet is always Unknown
runtimePolicyHostMetadata:
  enabled: true
env:
  SYRF__RuntimeFeatureFlags__Mode: "Shared"
  SYRF__RuntimeFeatureFlags__SharedReaderMode: "Disabled"
  SYRF__RuntimeFeatureFlags__Binding__Environment: "<env>"
  SYRF__RuntimeFeatureFlags__Binding__Database: "<database>"
  SYRF__RuntimeFeatureFlags__Binding__Cluster: "<cluster-fingerprint>"
  # Production only, API and PM:
  SYRF__RuntimeFeatureFlags__ProductionPolicyMode: "Runtime"
  # API only (runbook 6):
  SYRF__RuntimeFeatureFlags__FleetExpectedReplicas__api: "<api-replicas>"
  SYRF__RuntimeFeatureFlags__FleetExpectedReplicas__project-management: "<pm-replicas>"
  SYRF__RuntimeFeatureFlags__FleetInventoryMode: "Collector"

In production, Mode: Shared with ProductionPolicyMode: Legacy (the default) resolves to Legacy, so both are needed. Production also requires RuntimeEnvironment and ActiveEnvironment to be production. 2. R3.2 Review the rendered Deployments in the PR diff: four SYRF__RuntimeFeatureFlags__ host metadata entries from the chart, and the raw env: entries above. Raw env: cannot override the four host metadata entries. 3. R3.3 Merge and sync API and PM together, restoring the API replica count. This ends the API outage. 4. R3.4 Resume publishing, then check every host (preflight below), then end the freeze. With the continuous collector (R4.7) enabled, check its published log lines for both services (pod Ready only means the loop is stepping); nothing else is needed. Without it (or while it is being repaired), run the documented capture and publication (R4.3–R4.4) for both services by hand, in a loop whose full cycle, including the pause, stays under 30 seconds: each observation must be under 30 s old when the fleet is evaluated. Time one capture and publication per service before relying on the loop. If both together take more than about 20 s, drop the pause or run one loop per service. The previous IDs are the last accepted observation IDs from R4.5. --expected-observation-id is required here; only a service's first-ever publication omits it (R4.3). Shape (bash, one terminal per environment):

#!/usr/bin/env bash
set -euo pipefail
declare -A previous=([api]="<last-api-id>" [project-management]="<last-pm-id>")
declare -A deployment=([api]="<api-deployment>" [project-management]="<pm-deployment>")
declare -A container=([api]="<api-container>" [project-management]="<pm-container>")
while true; do
  for service in api project-management; do
    out="observations/${service}-$(date -u +%Y%m%dT%H%M%SZ).json"
    python3 scripts/capture-runtime-workload.py --context "<reviewed-context>" --namespace "<namespace>" \
      --service "$service" --deployment "${deployment[$service]}" --container "${container[$service]}" \
      --environment "<env>" --binding-epoch "<binding-epoch>" \
      --expected-observation-id "${previous[$service]}" --output "$out"
    receipt="$(SYRF_RUNTIME_COLLECTOR_TOKEN="<collector-token>" python3 scripts/publish-runtime-workload.py \
      --api-origin "<api-origin>" --environment "<env>" --binding-epoch "<binding-epoch>" \
      --service "$service" --input "$out")"
    previous[$service]="$(python3 -c 'import json,sys; print(json.loads(sys.argv[1])["observationId"])' "$receipt")"
  done
  sleep 2  # counts towards the 30 s budget
done

The loop stops on the first non-zero exit. On publish exit 3, replay the same $out file before capturing again (R9.3); on exit 1 or 2, fix the cause (runbook 4 refusals) and restart with the last accepted IDs. Tokens expire: obtain a new one (R4.2) before it does. Never put the token in a file or in shell history.

Preflight checks and expected output

Check	Expected output
API start-up	No InvalidOperationException from authority options (see refusals). /health/live healthy on every pod
GET /api/runtime-feature-flags/authority-status	mode: Shared, state: Ready, identity.environmentId: <env>, identity.bindingEpoch: <binding-epoch>, identity.storeEpoch equal to the receipt's
GET /api/runtime-feature-flags/authority-evaluation/disableMembership	state: Evaluated, value equal to the baseline or saved override, safeValue: false, productionEligible: true, maximumAgeSeconds: 30
GET /api/runtime-feature-flags/disableMembership/fleet	Until the loop has published fresh observations of both services: state: Unknown with a Collector… reason (for example CollectorInventoryMissingOrInvalid or CollectorInventoryStaleOrInvalidClock). This is expected. Then state: Active within 60 s, one host row per expected pod of each service, each state: Current; rows of pods that no longer exist show Retired
GET /api/runtime-feature-flags (anonymous snapshot)	200, with an authority envelope. A 503 with runtime_policy_unavailable means no current authority
Legacy PUT PUT /api/runtime-feature-flags/<key>	403 Runtime changes are not available in this mode. This is expected in Shared mode

PM has no admin authority routes. Its adoption shows only in the fleet host list.

Evidence to capture

• The cluster-gitops PR URL, approval and the rendered env diff.
• The authority-status, authority-evaluation and fleet responses after rollout.
• The end time of the freeze.

If a step is refused

Start-up refusals stop the host before it serves:

Start-up message	Cause
RuntimeFeatureFlags:Mode must be Legacy, Shadow or Shared.	Bad mode value
Shared authority mode cannot also enroll the legacy-format shared reader.	SharedReaderMode is not Disabled
Authority observation requires matching environment identities and explicit production Runtime admission.	RuntimeEnvironment/ActiveEnvironment differ, or production without Runtime admission
Authority observation requires explicit binding expectations: …	A binding value is missing or invalid
Production Runtime requires Shared mode and both production environment identities.	Production ProductionPolicyMode=Runtime without Mode=Shared
ProductionPolicyMode must be Legacy or Runtime.	Bad value
runtimePolicyHostMetadata requires boolean hpa.enabled (render) and similar	Fix the chart values
Legacy runtime policy provider cannot read the runtime policy document. Inspect the document before restarting this host.	The legacy store could not read the global document at all. Inspect it (read-only) and stop. An imported authority is not this case: a Legacy API starts against it and serves the deployed values (runbook 10)
Legacy runtime policy provider cannot serve initialized authority. Deploy the authority-capable adapter before restarting this host.	Only an image older than #4414, started in Legacy mode against the imported authority. Finish R3.1, or deploy a current image. The refused host has already written a syrf-api writer-registry entry (R2.7)

After start-up, authority-status Quarantined or Unavailable and fleet states other than Active are handled in runbooks 6 and 9.

4. Collector identity and scope

Purpose

Give the environment one workload collector that publishes API and PM Deployment and pod facts to POST /api/environment/observations, using a machine identity whose only scope is environment:observations:write. Collector evidence is what lets reports become Complete and what Collector inventory mode reads.

Preconditions

• Migration 0018 is applied.
• A read-only Kubernetes identity with get on the named Deployment and list on ReplicaSets and Pods in the namespace. For the manual path (R4.3–R4.5) this is the operator's reviewed context; staging has the precedent syrf-stats-preflight. The continuous collector (R4.7) brings its own: the API chart renders a namespace Role with exactly those verbs, bound to the collector's ServiceAccount. TO CONFIRM (OD-23, owner decision): whether production still needs the camarades-infrastructure change now that the chart renders the Role through GitOps.
• A machine client for the scope:
• OpenIddict environments: Identity seeds syrf-environment-collector only from an explicit strong EnvironmentCollector:ClientSecret (at least 32 characters);
• production: an Auth0 machine application with the same scope name, a production-shared tenant change that needs Chris's approval.
• Provision a dedicated namespace Secret. Set Identity chart environmentCollector.enabled: true and environmentCollector.clientSecret.secretName and .key to its reference. The default key is clientSecret. The continuous collector (R4.7) mounts the same Secret reference.
• Capture and publication run continuously in the cluster with the API chart's runtimeWorkloadCollector values (R4.7, #4748). The values default to off, and no environment has them enabled. The manual steps R4.3–R4.5 stay the way to prove the first publication and to recover.

Steps

1. R4.1 Configure the API (raw env:, API values):

env:
  SYRF__RuntimeFeatureFlags__WorkloadCollector__Enabled: "true"
  SYRF__RuntimeFeatureFlags__WorkloadCollector__Provider: "<OpenIddict|Auth0>"
  SYRF__RuntimeFeatureFlags__WorkloadCollector__ClientId: "<collector-client-id>"
  SYRF__RuntimeFeatureFlags__WorkloadCollector__Audience: "<api-audience>"

Audience must equal IdentityService:Audience (OpenIddict) or Auth0:ApiAudience (Auth0). The SYRF__RuntimeFeatureFlags__Binding__* values (as in R3.1) must also be set, and <runtime-env> must match them. The collector does not need Shared mode, so this can run before enrolment; Legacy mode ignores the binding values. 2. R4.2 Obtain a token (OpenIddict), the same form as scripts/fetch-statistics-parity.py:

POST <identity-origin>/connect/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&client_id=<collector-client-id>&client_secret=<secret>&scope=environment:observations:write

TO CONFIRM: the token request for the production Auth0 machine application (audience parameter). 3. R4.3 Capture one service, read-only, with an explicit reviewed context:

python3 scripts/capture-runtime-workload.py \
  --context "<reviewed-context>" --namespace "<namespace>" \
  --service api --deployment "<api-deployment>" --container "<api-container>" \
  --environment "<env>" --binding-epoch "<binding-epoch>" \
  --expected-observation-id "<last-accepted-observation-id>" \
  --output api-observation.json

The chart default Deployment and container name is the chart name (syrf-api, syrf-projectmanagement); check the rendered manifest. Omit --expected-observation-id only for a known first publication. 4. R4.4 Publish it at once (publication refuses snapshots older than 30 seconds):

SYRF_RUNTIME_COLLECTOR_TOKEN="<collector-token>" python3 scripts/publish-runtime-workload.py \
  --api-origin "<api-origin>" --environment "<env>" \
  --binding-epoch "<binding-epoch>" --service api --input api-observation.json

1. R4.5 Repeat R4.3–R4.4 for project-management. Keep each service's last accepted observation ID.
2. R4.6 Keep publishing. Collector inventory needs observations strictly younger than 30 seconds for both services, so both must be republished well inside 30 seconds for as long as Collector-mode Active is needed. Use the continuous collector (R4.7). The manual loop in R3.4 is the fallback.

3. R4.7 Run the continuous collector (API chart, runtimeWorkloadCollector, default off). One pod per environment runs scripts/run-runtime-workload-collector.py. Every 10 seconds, for API and then PM, it takes a client-credentials token for environment:observations:write, captures the Deployment in-cluster with its own ServiceAccount (capture-runtime-workload.py --in-cluster) and publishes the exact bytes at once (publish-runtime-workload.py). Each publication chains on that service's last accepted observation ID.

4. Prerequisites: R4.1 is deployed. Identity seeds the collector client from the namespace Secret (preconditions). R4.3–R4.5 succeeded once by hand, and you have each service's last accepted observation ID.

5. Prepare one cluster-gitops change for the API service values (shape; never a secret value):

   runtimeWorkloadCollector:
     enabled: true
     apiOrigin: "<api-origin>"                      # HTTPS origin, no path
     tokenUrl: "<identity-origin>/connect/token"    # OpenIddict; Auth0: TO CONFIRM (gap 9)
     clientId: "<collector-client-id>"              # default syrf-environment-collector
     audience: ""                                   # Auth0 only
     environment: "<env>"
     bindingEpoch: "<binding-epoch>"
     expectedNamespace: "<namespace>"               # must equal the release namespace
     clientSecret:
       secretName: syrf-environment-collector       # the Secret Identity's environmentCollector reads
       key: clientSecret
     seedObservationIds:                            # last accepted IDs from R4.5; used only on an empty state
       api: "<last-api-id>"
       projectManagement: "<last-pm-id>"
   

   deployments.api and deployments.projectManagement default to syrf-api and syrf-projectmanagement for both the Deployment and container names. Check them against the rendered manifests. 3. Review the render in the PR diff. Expect these resources: - Deployment syrf-api-workload-collector: 1 replica, Recreate. The image is python:3.13.18-alpine3.24 pinned by digest. The pod runs non-root with a read-only root filesystem and automountServiceAccountToken: true. - ServiceAccount: automountServiceAccountToken: false. - Role: get on the two named Deployments and list on replicasets and pods, nothing else. - RoleBinding to the ServiceAccount. - ConfigMap with the three scripts. - PersistentVolumeClaim syrf-api-workload-collector-state (1 Gi).

   The render fails without apiOrigin, tokenUrl, environment, bindingEpoch or a matching expectedNamespace. The staging and production AppProjects already allow each of these kinds (cluster-gitops argocd/projects). 4. Merge and sync. Then check the preflight rows for R4.7 below.

What the loop does with each result:

Result	Loop action
Publish exit 0	Saves the ID as the service's chain head. Logs published (or replayed) with the observation ID, digest and timings
Publish exit 3 (408/425/429, 5xx, transport loss)	Keeps the exact file. Replays it before capturing anything new for that service (outcome-unknown, then replayed). A replay refused with 409 is treated as never stored, so the loop captures again from the last accepted ID. Its refused line carries seedCandidate: true and the observationId: if the original may have been stored (409 can also mean publication disabled or bound elsewhere) and later ticks log chain-refused, use that ID as a seed
HTTP 409 on a fresh capture	Captures again once with the last accepted ID. If refused again: chain-refused, and the tick fails
HTTP 401/403	Drops the token and keeps the capture for replay (unauthorized)
HTTP 400 or another refusal	Discards the capture (refused)
Capture Workload changed during capture	Captures again, up to three captures per service per tick
Any other capture refusal	capture-refused, with the sanitized reason. The other service still publishes

A failed tick retries after 1, 2, 4 … seconds, capped at maxBackoffSeconds (30). The chain head and any capture awaiting replay are written to the state volume before each send, so a restarted or replaced pod replays and continues the chain instead of forking it. Logs carry IDs, digests, timings and HTTP statuses only. SIGTERM finishes the current step, saves the state and exits 0.

Stop or retire. Set runtimeWorkloadCollector.enabled: false (cluster-gitops). Both the staging and production API Applications run with automated prune (syncPolicy.automated: {prune: true, selfHeal: true}, read on 4 November 2026), so this prunes the pod, its RBAC and its state claim. The fleet then falls to Unknown (Collector…) within 30 seconds, as with any collector outage. To enable it again, re-run R4.3–R4.5 or read the last accepted IDs, and set them as seedObservationIds.

Preflight checks and expected output

Check	Expected output
R4.3 capture	Exit 0, api-observation.json created; never overwritten
R4.4 publish	Exit 0; stdout only {"observationId": "<observation-id>", "digest": "<64 hex>"}
After R4.5: report (runbook 7, R7.3)	Sources workloads:api and workloads:project-management are Current
After R4.5: Collector-mode fleet (runbook 6)	state: Active; replaced pods' rows show Retired
R4.7 collector pod	Running and Ready, and published log lines for both services about every 10 s. Ready and liveness only mean the loop is stepping (readiness never waits for a publication, so a failing collector never holds the API Application Progressing or its PostSync notification)
R4.7 collector log (read-only)	One started line with the binding, deployments and the chain heads it loaded, then a published line per service about every 10 s, each with observationId, expectedObservationId equal to the previous one, digest, captureMs, publishMs and ageMs
R4.7 after a rollout of API or PM	A new published observation listing the new pods; the fleet's replaced rows become Retired (runbook 6)

Evidence to capture

• The client registration change and its approval (no secret values).
• Each capture file and its publish receipt (observation ID and digest).
• The Kubernetes identity's role definition reference.
• For R4.7: the cluster-gitops PR, the rendered Role and the collector's started and first published log lines.

If a step is refused

Result	Meaning and action
Capture exit 2 Capture refused: Output already exists; preserve it for exact replay	Use a new output path; keep the old file
Capture exit 2 Capture refused: Workload changed during capture; acquire a new observation	A rollout is in progress; capture again
Capture exit 2 Capture refused: Capture clock or freshness budget invalid	Reads took over 20 seconds or the clock moved back
Capture exit 2 Capture refused: Observation exceeds publication body limit	Over 512 KiB; investigate the pod count
Publish exit 2 Publication refused locally; check origin, binding, capture and credential inputs	Fix the inputs. HTTPS only, no path; capture must match --environment, --binding-epoch, --service
Publish exit 1 Publication refused: HTTP 400	Invalid workload observation: contract failure, for example a pod without explicit ready/terminating
Publish exit 1 Publication refused: HTTP 409	Workload observation refused. Causes in the store: expected observation missing or not advanced, an observation ID reused with different evidence, a stale or future snapshot, a changed binding, a missing 0018 index, or publication disabled or bound elsewhere. Capture again with the right previous ID
Publish exit 1 HTTP 401 or HTTP 403	Token scope, client, audience or binding mismatch, or the collector is disabled
Publish exit 3 Publication outcome unknown; preserve and replay the exact capture	408/425/429, 5xx (503 Workload publication outcome unknown…) or transport loss. Replay the same file
API start-up Enabled workload collection requires explicit matching provider, client, audience and environment binding.	Fix R4.1
R4.7 render fails runtimeWorkloadCollector.… is required when the collector is enabled (or … does not match the release namespace)	Fix the values in the cluster-gitops change
R4.7 pod exits 2 Collector refused: …	Start-up refusal, nothing sent. Examples: In-cluster ServiceAccount belongs to another namespace, State belongs to another environment or binding; keep it and review before clearing, A capture awaiting replay is missing; recover the chain head before restarting, Another collector holds the state directory
R4.7 log chain-refused on every tick, no published lines	The stored chain head is not the one this collector holds. Typical cause: the state claim was lost, or the seeds are stale. Read the last accepted IDs (report, runbook 7 R7.3), delete the claim by disabling and re-enabling the collector, and set seedObservationIds to them in the same change
R4.7 log token-unavailable (Token request refused: HTTP 401) or unauthorized	The client secret, client, audience or scope does not match Identity or the API (R4.1)
R4.7 log capture-refused	The capture refusals above. A persistent Kubernetes read unavailable or invalid means the Role or ServiceAccount token is wrong, or the TLS check against the mounted cluster CA failed. The image's Python 3.13 verifies X.509 strictly: the CA needs key usage, and the server certificate needs key identifiers. The isolated proof hit this with a throwaway CA. TO CONFIRM at staging enablement that GKE's cluster CA passes

5. Coordinator identity

Purpose

Give the promotion workflow a machine identity, scope environment:deployments:coordinate, that records, extends and releases one deployment compatibility constraint per rollout through PUT and DELETE /api/environment/deployment-constraints/<rollout-id>.

Preconditions

• The API is in Shared mode with the binding (runbook 3).
• A machine client:
• OpenIddict: Identity seeds syrf-environment-coordinator only when EnvironmentCoordinator:ClientSecret has at least 32 characters;
• production: an Auth0 machine application (tenant change, Chris's approval).
• TO CONFIRM: no chart value renders EnvironmentCoordinator:ClientSecret into Identity.
• TO CONFIRM: no workflow calls the coordinator routes yet, and how the coordinator builds the capability intersection and completion targets from a trusted report is not implemented.

Steps

1. R5.1 Configure the API (raw env:):

env:
  SYRF__RuntimeFeatureFlags__DeploymentCoordinator__Enabled: "true"
  SYRF__RuntimeFeatureFlags__DeploymentCoordinator__Provider: "<OpenIddict|Auth0>"
  SYRF__RuntimeFeatureFlags__DeploymentCoordinator__ClientId: "<coordinator-client-id>"
  SYRF__RuntimeFeatureFlags__DeploymentCoordinator__Audience: "<api-audience>"

1. R5.2 Record a constraint before a rollout. Always send action; an omitted action means Record:

PUT /api/environment/deployment-constraints/<rollout-id>
Authorization: Bearer <coordinator-token>
Content-Type: application/json

{
  "mutationId": "<mutation-id>",
  "expectedIdentity": <identity>,
  "action": "Record",
  "reason": "<rollout reason>",
  "expiresAtUtc": "<future time>",
  "servingCatalogDigests": ["<serving-catalog-digest>"],
  "targetCatalogDigest": "<target-catalog-digest>",
  "allowedCapabilities": { "policySchemaVersion": 1, "enabledKeys": ["<key>"] },
  "completionTargets": {
    "target":   { "catalogDigest": "<target-catalog-digest>",  "serviceImages": { "api": "<repo>@sha256:<digest>", "project-management": "<repo>@sha256:<digest>" } },
    "rollback": { "catalogDigest": "<serving-catalog-digest>", "serviceImages": { "api": "<repo>@sha256:<digest>", "project-management": "<repo>@sha256:<digest>" } }
  }
}

1. R5.3 Extend with action: "Extend", the reviewed expectedConstraintDigest and a later expiresAtUtc.
2. R5.4 Release after the collector shows every replica on the target (or a completed rollback):

DELETE /api/environment/deployment-constraints/<rollout-id>
Authorization: Bearer <coordinator-token>
Content-Type: application/json

{ "mutationId": "<mutation-id>", "expectedIdentity": <identity>, "action": "Release",
  "reason": "<reason>", "expectedConstraintDigest": "<constraint-digest>", "completionKind": "Target" }

Clients and proxies must keep the JSON body on DELETE.

Preflight checks and expected output

Check	Expected output
Before R5.2: report (purpose=promotion)	deploymentConstraint source Current; projection null (no active constraint)
R5.2–R5.4 response	200 {"mutationId": "<mutation-id>", "rolloutId": "<rollout-id>", "action": "<action>", "constraintDigest": "<digest or null after release>", "recordedAtUtc": "<time>"}
After R5.2: report	deploymentConstraint.state: Active, rolloutId, constraintDigest

Evidence to capture

• Each request and receipt (mutation ID, rollout ID, constraint digest).
• The report before record and after release.

If a step is refused

Response	Meaning and action
400 Invalid coordinator operation	Wrong verb for the action (Record/Extend need PUT, Release needs DELETE) or a rollout ID that is not one ASCII segment of up to 128 [A-Za-z0-9-_.:]
400 Invalid coordinator request	Malformed identity or fields
409 Policy identity changed; read a fresh report	Revalidate against the new policy, then start again
409 Deployment constraint operation refused	The store refused. Possible causes: another active constraint, policy outside the capability intersection, missing completion expectations, unproven completion, not the owning coordinator, expiry not moved forward. The response does not name the cause
503 Coordinator outcome unknown; retry the exact mutation ID and request	Retry unchanged
401/403	Wrong scope, client, audience or binding; coordinator disabled
API start-up Enabled deployment coordination requires explicit matching provider, client, audience and environment binding.	Fix R5.1

A stranded constraint is released by an administrator through break-glass (runbook 9). An expired constraint keeps blocking incompatible changes; expiry is never a release.

6. Expected inventory and Collector mode

Purpose

Tell the API how many API and PM hosts must acknowledge a policy before the fleet is Active, and let it read that membership from fresh collector observations instead of every stored acknowledgement row.

Preconditions

• Runbook 3 (hosts publish acknowledgements) and runbook 4 (fresh observations for both services).
• Fixed replica counts and hpa.enabled: false for API and PM.
• For production enables and restores: Collector mode with the collector publishing (see Status and boundary). Static mode refuses them after any rollout.

Steps

1. R6.1 Set the per-host metadata through the chart (runtimePolicyHostMetadata.enabled: true). Each pod renders InstanceUid from metadata.uid, ImageVersion from the exact image, ExpectedReplicas from replicaCount and Autoscaled from hpa.enabled.
2. R6.2 Set the fleet inventory on the API: SYRF__RuntimeFeatureFlags__FleetExpectedReplicas__api and SYRF__RuntimeFeatureFlags__FleetExpectedReplicas__project-management, each equal to that service's replicaCount. No count is inferred from observed hosts.
3. R6.3 Set SYRF__RuntimeFeatureFlags__FleetInventoryMode: "Collector" on the API. The default is Static. Any other value refuses inspection.
4. R6.4 Change counts only together. A replica change edits replicaCount and the matching FleetExpectedReplicas value in the same cluster-gitops change.

Notes:

• replicaCount: 0 renders ExpectedReplicas: "0" as metadata only; a zero-count host publishes no acknowledgement. FleetExpectedReplicas must be positive, so a zero-count service leaves the fleet Unknown (ExpectedInventoryMissingOrInvalid).
• Acknowledgements count as current only if younger than 30 seconds. Hosts poll every 8–12 seconds, and 6B measured adoption up to 11.5 seconds. That margin is close to the limit; treat brief Applying as normal.
• Static mode suits only hosts whose instance UIDs stay the same (the local harness). In Kubernetes every replaced pod has a new UID, and old rows have no TTL, so a Static fleet reads Applying, then Degraded, after every rollout.
• In Collector mode a row absent from fresh collector inventory is retired only once MongoDB has not seen it rewritten for the 35-second quiet period (30-second freshness plus 5 seconds, server time; #4407). So a rollout reaches Active about 35 seconds after the old pods stop. Rows with no server stamp, written by images older than #4407, need 15 minutes, the assumed bound on host clock skew, so the first rollout away from those images can take up to 15 minutes.

Preflight checks and expected output

GET /api/runtime-feature-flags/disableMembership/fleet (Cache-Control: no-store):

Fleet state / reason	Meaning
Active / null	Every expected host of both services is Current on this identity
Saved or Applying / HostsUnconfirmed	Within the 60-second confirmation window
Degraded / HostsUnconfirmed	Window passed with hosts missing or behind
Unknown / ExpectedInventoryMissingOrInvalid	A FleetExpectedReplicas value is missing, non-positive or names another service
Unknown / AutoscaledInventory	A host reports Autoscaled
Unknown / DomainReadinessNotEstablished	Any key without a new-request-admission consumer contract (only disableMembership and the targeting pilot screeningKeywordHighlighting have one)
Unknown / Collector… reasons	CollectorInventoryMissingOrInvalid, CollectorBindingMismatch, CollectorInventoryStaleOrInvalidClock, CollectorExpectedCountDisagrees, CollectorRolloutIncomplete, CollectorHostServiceDisagrees, CollectorHostObservationDisagrees
Action required / ExpectedInventoryDisagrees, DuplicateInstanceIdentity, IncompatibleHost, InvalidCommitClock	Fix the configuration or the incompatible host

Host rows carry state Current, Behind (PolicyNotApplied), Unknown (HostMetadataMissing, StaleOrInvalidObservationClock, or the host's error class), Incompatible (IdentityOrCatalogMismatch) or, in Collector mode, Retired (AbsentFromFreshCollectorInventory). Retired rows stay stored and visible.

Evidence to capture

• The values change and the fleet response showing Active with the expected host count.
• In Collector mode, a fleet response after a rollout showing old rows Retired.

If a step is refused

• 403 Fleet inspection requires Shared API authority mode.: runbook 3 is not complete on this API.
• 503 runtime_policy_unavailable: the snapshot could not be read; retry.
• An unsupported FleetInventoryMode value refuses inspection (Unsupported fleet inventory mode.).

7. Change one flag

Purpose

Make one reviewed policy change to <key> (in production, disableMembership only): preview it, commit exactly the previewed change, then confirm that every expected host has adopted it.

Preconditions

• Runbooks 3 and 6 complete; fleet Active for the current identity.
• The operator holds ChangeRuntimeFeaturePolicy (Administrator, OD-2).
• For production enables: Collector mode with the collector publishing, the release guidance (OD-26) and Chris's approval.
• The admin page's preview/commit UI (#4381) is not on main; use the API routes.

Steps

1. R7.1 Read the identity (authority-status, above). Expect state: Ready.
2. R7.2 Check the fleet (GET /api/runtime-feature-flags/<key>/fleet). Expect state: Active.
3. R7.3 Check compatibility. Collect a report with the report token, save it, then run the checker:

GET /api/environment/compatibility-reports/current?purpose=policy-change
Authorization: Bearer <report-token>

python3 scripts/runtime-compatibility-check.py --report report.json --candidate candidate.json \
  --environment "<env>" --binding-epoch "<binding-epoch>" --purpose policy-change

The candidate is the serving build (syrf.runtime-candidate/v1). Produce candidate.json with that API image's read-only candidate mode (Producing the candidate). It reads no input and needs no database, binding or cluster access, so it runs anywhere the image can be pulled:

docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/out" \
  -e SYRF_RUNTIME_POLICY_MODE=candidate -e SYRF_RUNTIME_POLICY_OUTPUT=/out/candidate.json \
  -e SYRF_RUNTIME_CANDIDATE_TARGET_MODE=Runtime \
  -e SYRF_RUNTIME_CANDIDATE_IMAGE_API="<api-imageVersion>" \
  -e SYRF_RUNTIME_CANDIDATE_IMAGE_PM="<pm-imageVersion>" \
  ghcr.io/camaradesuk/syrf-api:<serving-tag>

--user lets the image's non-root process write to the mounted directory; without it the write fails with exit 1. Copy each image from the report's applied[].imageVersion for that service (<repository>:<tag>). Exit 0 means the file was written; exit 2 is a refusal (Runtime policy candidate refused: …, nothing written); exit 1 leaves no file, so run it again. Never edit the file. Never pass --now for a real decision. Reports need RuntimeFeatureFlags:CompatibilityReport settings (Enabled, Provider, ClientId, Audience) and the client syrf-environment-report with a separate namespace Secret. Set Identity chart environmentReport.enabled: true and point environmentReport.clientSecret.secretName and .key at that credential. 4. R7.4 Preview (read-only, allowed under support impersonation):

POST /api/runtime-feature-flags/<key>/preview
<admin-session>
Content-Type: application/json

{
  "mutationId": "<mutation-id>",
  "expectedIdentity": <identity>,
  "value": true,
  "reason": "<reason>",
  "action": "Change"
}

value is true, false or null (clear the override). If the response is 422 featureFlagDependenciesRequired, review the set and repeat with "confirmedDependencies": {…}, the exact key/value set. Keep includeDependencies false. 5. R7.5 Commit the same body plus the preview digest:

PUT /api/runtime-feature-flags/<key>/policy
<admin-session>
Content-Type: application/json

{ …the R7.4 body unchanged…, "previewDigest": "<preview-digest>" }

1. R7.6 Confirm adoption. Poll the fleet until state: Active and every host row shows the committed revision with state: Current. Check authority-evaluation/<key> on the API for the new revision and value. Browsers are not part of adoption. The commit wakes the writing pod's own refresh, so its tabs get a RuntimeFeatureFlagsRevision hint as soon as that pod's reader has the new identity (one authority read, bounded by its 5 s deadline); tabs on other API pods get one when that pod's next refresh (8–12 s) observes it. Either way the tab refetches the snapshot. A failed hint is only a log line (Runtime policy revision … hint failed or … hint unavailable); it never changes the receipt.
2. R7.7 If the response was lost, look up the receipt, then retry the unchanged request if needed:

GET /api/runtime-feature-flags/mutations/<mutation-id>
<admin-session>

Preflight checks and expected output

Check	Expected output
R7.3 report	completeness: Complete, purpose: policy-change, every entry of requiredSources Current. In Static mode reports stay Incomplete (fleetInventoryMode:Static in unknownOrUnsupported)
R7.3 checker	{"exitCode": 0, "reasons": [], "verdict": "pass"}, exit 0
R7.4 preview	200 with previewDigest, before (the current identity), proposed (revision + 1), changes containing {"key": "<key>", "proposedOverride": true, "beforeEffective": false, "proposedEffective": true, "cause": "Requested", …} and fleetReadiness: the current fleet state in production (Active needed to enable), Unknown elsewhere
R7.5 commit	200 {"mutationId": "<mutation-id>", "identity": {…, "revision": "<revision + 1>"}}. Committed, not yet adopted
R7.6 fleet	Active for the new identity within the 60-second window; 6B measured adoption up to 11.5 seconds
R7.7 lookup	{"mutationId": "<mutation-id>", "outcome": "Committed", "committedIdentity": {…}}, or outcome: Unknown with no identity, which does not prove an abort

Evidence to capture

• The report ID, the report file and the checker output.
• The preview response, the commit receipt and the fleet responses until Active, with timestamps.

If a step is refused

Response	Meaning and action
403 (no body)	The caller lacks ChangeRuntimeFeaturePolicy
503 authority-unavailable	Application authority could not decide; retry after Retry-After. During a rollout, see D4
403 Policy administration requires Shared API authority mode.	This API is not enrolled
409 runtime_policy_revision_conflict with actualRevision	Someone changed the policy. Re-read and decide again
422 featureFlagDependenciesRequired	Confirm dependencies (R7.4). canApplyAutomatically: false means a deployment change is needed first
400 runtime_policy_constraint, message: The reviewed preview is missing or stale. Preview the current policy and retry.	Preview again; the digest no longer matches
… Production enable or restore requires fresh fixed-fleet acknowledgement of the current policy.	Fleet not Active. In Static mode this follows every rollout; set Collector mode and keep the collector publishing (runbook 6)
… This key is not eligible for production runtime policy.	Only disableMembership
… A production plan cannot change ineligible dependencies or safety pins.	The change would touch other keys in production
… Policy requires a capability outside the deployment intersection.	An active deployment constraint forbids it; wait for release
… Emergency stops use their independent safe-state path.	Use runbook 8
… Authority mutation requires an exact verified environment binding., Environment binding changed before the authority transaction., Authority identity does not match the verified binding., The request belongs to a different authority lineage.	Runbook 9
… Mutation ID was already used with a different request.	Use a new mutation ID for a new request
503 runtime_policy_commit_unknown	Retry the same mutation ID and unchanged request (R7.7)
503 runtime_policy_unavailable	Preview or read failed; nothing was written
Checker exit 1 or 2	See the reason codes in Checking a candidate offline

8. Emergency stop and restore

Purpose

Force an audited global safe-false control to its safe value at once, keeping the saved baseline and overrides, and later restore the saved policy through a reviewed preview and commit.

Preconditions

• The API is in Shared mode. The operator holds EmergencyStopRuntimeFeatures for a stop and ChangeRuntimeFeaturePolicy for the restore.
• A stop needs no fleet readiness, deployment-constraint clearance, content or new audit indexes. It still needs the binding, lineage and receipt integrity.
• A production restore that makes the control effective On needs the fleet Active on the current (stopped) policy, so the same Collector-mode and running-collector prerequisite applies.

Steps

1. R8.1 Read the identity (authority-status).
2. R8.2 Stop:

POST /api/runtime-feature-flags/<key>/emergency-stop
<admin-session>
Content-Type: application/json

{ "mutationId": "<mutation-id>", "expectedIdentity": <identity>, "reason": "<reason>" }

1. R8.3 Confirm the stop reached every host: authority-evaluation/<key> shows value: false at the stop revision on the API, and the fleet host rows show the stop revision Current. A 503 or lost response is retried with the same mutation ID and body.
2. R8.4 Restore when the incident is resolved. Read the identity again, then preview and commit ReleaseStop with a null value and no dependency changes:

POST /api/runtime-feature-flags/<key>/preview
<admin-session>
Content-Type: application/json

{ "mutationId": "<new-mutation-id>", "expectedIdentity": <identity>, "value": null,
  "reason": "<reason>", "action": "ReleaseStop" }

Then PUT /api/runtime-feature-flags/<key>/policy with the same body and previewDigest. The local harness sends action as its ordinal (2). The API reads request bodies with Newtonsoft.Json, which also accepts the name: the 6C.2 rehearsal sent "ReleaseStop" (and "Change" in R7.4) as names and both were accepted. 5. R8.5 Confirm adoption of the restore revision as in R7.6.

Preflight checks and expected output

Check	Expected output
R8.2 response	200 {"mutationId": "<mutation-id>", "identity": {…}}: durably committed, not yet adopted
R8.3	Every API evaluates false at the stop revision; the 6B harness checks this within 60 seconds with a PM host paused
R8.4 preview	The changes entry shows wasStopped: true, willBeStopped: false and proposedOverride equal to the saved override
R8.5	Every API evaluates the restored saved value; fleet Active

A new mutation ID always records a new revision, even if the key is already stopped. Idempotency is per mutation ID.

Evidence to capture

• The reason, the stop receipt and the per-host adoption times.
• The restore preview, receipt and fleet Active response.

If a step is refused

Response	Meaning and action
403 (no body)	The caller lacks EmergencyStopRuntimeFeatures
403 Emergency stops require Shared API authority mode.	Not enrolled
409 runtime_policy_revision_conflict	Re-read the identity and decide; a stop is often still wanted
400 runtime_policy_constraint: Stop actions require an audited global safe-state contract and cannot change saved values or dependencies.	The key has no audited safe-false contract
503 runtime_policy_commit_unknown	Retry the same mutation ID and request
Restore refusals	As runbook 7, including the production fleet refusal

9. Recovery and rebind

Purpose

Recover from the failure cases the code reports, without bypassing a binding, a lineage or an audit record. Every recovery is a forward action.

Preconditions

• The failure is identified from a response, log line or state below.
• Approval as for the original step. Anything that would edit the binding, the authority document or the audit by hand needs Chris and is not covered here.

Steps

1. R9.1 Lost or unknown policy outcome. GET /api/runtime-feature-flags/mutations/<mutation-id>. On Committed, record it and stop. On Unknown, retry the unchanged original request.
2. R9.2 Lost lineage apply outcome. Rerun R2.5 with the identical preview and a new output path.
3. R9.3 Unknown workload publication (exit 3). Replay the exact capture file. Do not change evidence under an existing observation ID.
4. R9.4 Mongo outage. No operator action. Decisions fail closed to the safe value on the first failed read and recover by themselves after the outage (6B mongo-outage scenario). Confirm with R7.6 afterwards.
5. R9.5 Reader quarantine. authority-status shows Quarantined (InvalidBinding, IdentityOrRevisionRegression, or AuthorityNotInitialized after a snapshot). There is no automatic reset. Find out why the identity, binding or revision changed, then restart the affected pods with a reviewed cluster-gitops change. TO CONFIRM: the reviewed recovery procedure that must precede the restart is not written.
6. R9.6 Stranded deployment constraint. A real administrator (no impersonation, no machine scope) calls:

POST /api/environment/deployment-constraints/<rollout-id>/break-glass-release
<admin-session>
Content-Type: application/json

{ "mutationId": "<mutation-id>", "expectedIdentity": <identity>,
  "reason": "<reason, max 1000 characters>", "expectedConstraintDigest": "<64-character digest>" }

It needs BreakGlassRuntimeDeploymentConstraint and Shared mode. Expected: 200 with the coordinator receipt shape. 7. R9.7 Restored database or changed cluster (rebind). A restored copy carries the source binding, so up refuses with exit 2 and the reader quarantines. rebind is not built: no supported command replaces a binding. Stop and raise it with Chris. Do not edit runtimeEnvironmentBinding by hand.

Preflight checks and expected output

Check	Expected output
R9.1	outcome: Committed with committedIdentity, or Unknown
R9.4 after recovery	authority-evaluation state: Evaluated at the committed revision; fleet Active
R9.5 after restart	authority-status state: Ready with the expected identity
R9.6	200 {"mutationId", "rolloutId", "action", "constraintDigest": null, "recordedAtUtc"}; the report's constraint projection is null
R9.7	status shows the binding record and the live fingerprint, and the mismatch

Evidence to capture

• The failure response or log line, the lookup result and the retry receipt.
• For break-glass: the reason, the before-state report and the receipt.
• For R9.7: the status output and the escalation record.

If a step is refused

Response	Action
Mutation lookup 403 Mutation outcomes require Shared API authority mode.	Use a Shared-mode API
Mutation lookup 503	Storage failure: no outcome. Retry the lookup
Break-glass 403	Impersonation, a machine scope, more than one identity, or a missing capability
Break-glass 403 Break-glass requires Shared API authority mode.	Not enrolled
Break-glass 400 Invalid rollout identity	Fix the rollout ID
Break-glass 409 Policy identity changed; inspect current state / Deployment constraint recovery refused	Read the current policy and constraint, then decide again
Break-glass 503 Break-glass outcome unknown; retry the same mutation ID and unchanged request	Retry unchanged

10. Legacy return

A host in Legacy mode against an imported authority (owner decision D7=A, spec 6C.5) starts and serves the deployment-controlled values for every key. It never applies the authority's saved overrides or stops, never writes the authority document, refuses legacy changes, and registers as a read-only reader so it cannot block a later lineage freeze. The saved policy, baseline, receipts, audit and stops stay exactly as they were, so returning to Shared (or production Runtime) resumes the same identity. The legacy-return scenario of the runtime-fleet harness rehearses this runbook end to end on an isolated stack.

Purpose

Return API and PM from runtime policy (Shared, or Runtime in production) to deployment-controlled values without any change in behaviour (RT-15), and come back without losing the saved policy.

Preconditions

• No active stop on any production-eligible key.
• For every production-eligible key (today disableMembership), the runtime decision equals the deployed FeatureFlags value.
• Collector mode, so the report can be Complete.
• Chris's approval of the mode-switch change.

Steps

1. R10.1 Make the values agree. Either commit a reviewed forward revision so the runtime value equals the deployed value (runbook 7), or change the deployed value in GitOps (featureFlags.disableMembership) to the runtime value. Release any stop first (runbook 8).
2. R10.2 Collect a legacy-return report:

GET /api/environment/compatibility-reports/current?purpose=legacy-return
Authorization: Bearer <report-token>

1. R10.3 Run the checker with a candidate whose targetMode is Legacy (the build that will serve):

python3 scripts/runtime-compatibility-check.py --report report.json --candidate candidate.json \
  --environment "<env>" --binding-epoch "<binding-epoch>" --purpose legacy-return

Record the authority identity from authority-status (all seven fields) before the switch. 4. R10.4 Switch the mode for API and PM in one cluster-gitops change, changing nothing else: production SYRF__RuntimeFeatureFlags__ProductionPolicyMode: "Legacy"; elsewhere SYRF__RuntimeFeatureFlags__Mode: "Legacy". Saved policy, baseline, receipts, audit and stops are not changed by the switch. 5. R10.5 Confirm after the rollout, on each API: - authority-status returns mode: Legacy, state: Disabled, reason: LegacyModeAuthorityPreserved and no identity (a Legacy host never reads or verifies the authority); - the anonymous snapshot GET /api/runtime-feature-flags has policyState: DeploymentControlledAuthorityPreserved, mutationAllowed: false, no overrideValue on any flag, and disableMembership at the report's legacyValue; - provider-status has policyState: DeploymentControlledAuthorityPreserved and policyStateDetail saying the runtime policy is preserved but not applied; - the admin page shows "Deployment-controlled; runtime policy preserved" and offers no changes. 6. R10.6 Return to runtime policy when wanted: revert only the mode (production ProductionPolicyMode: "Runtime"; elsewhere Mode: "Shared") for API and PM together. authority-status returns Ready with the identity recorded in R10.3, unchanged in all seven fields, and the fleet reaches Active.

Preflight checks and expected output

Check	Expected output
R10.2 report	completeness: Complete, purpose: legacy-return, legacyParity.agrees: true, each entry stopActive: false, legacyValue equal to runtimeValue
R10.3 checker	{"exitCode": 0, "reasons": [], "verdict": "pass"}, exit 0
R10.5 legacy change	PUT /api/runtime-feature-flags/<key> (or /groups/<key>) answers 409 with code: runtime_policy_authority_preserved; nothing is written
R10.5 registry	db.syrf_metadata.findOne({_id: "meta"}, {"services.syrf-api.instances": 1}) (read-only) shows each Legacy API with runtimePolicyReadOnly: {reason: "LegacyReaderAuthorityPreserved", connectedAt: <its own connectedAt>}
R10.5 authority unchanged	The global document and the audit row count read before R10.4 are unchanged (read-only queries)
R10.6 return	authority-status Ready with the R10.3 identity; fleet Active

Evidence to capture

• The report ID and file, the candidate manifest and the checker output.
• The forward revision or GitOps change from R10.1.
• The identity before R10.4 and after R10.6, and the R10.5 outputs.
• The mode-switch PRs and their approvals.

If a step is refused

Checker reason	Exit	Action
legacy-parity-stop-active	1	Release the named key's stop (runbook 8)
legacy-parity-differs	1	R10.1 for the named key
legacy-parity-missing	2	The report has no bound policy; collect it while the hosts are still in Shared/Runtime
binding-mismatch	1	Wrong --environment/--binding-epoch, or a report from another environment
report-incomplete, required-source-unknown, report-expired, report-unstable, purpose-mismatch	2	Collect a fresh report with purpose=legacy-return in Collector mode
migration-not-applied, catalog-key-missing, catalog-digest-incompatible	1	The target build cannot honour the current state; plan a replacement release
Report 400 environment_report_purpose_invalid	n/a	purpose must be exactly policy-change, promotion or legacy-return, given once
Report 503 environment_report_unavailable	n/a	Missing binding, indexes or evidence; retry, then investigate

The checker exits 2 whenever any insufficient-evidence reason is present, even if refusal reasons are listed too. For example, a Static-mode report whose values differ gives exit 2 with both report-incomplete and legacy-parity-differs.

A Legacy host never refuses to start over the authority. If R10.5 shows a runtime value instead of the deployed one, the host is not in Legacy mode: check the rendered Mode/ProductionPolicyMode.

11. Historical release import (written only)

Written only. This runbook has not been run in any environment beyond the isolated tests. It is an operational prerequisite for showing past releases (for example the August 2026 What's New entries) with their real dates, and needs Chris's approval before any shared environment.

Purpose

Record a release that happened before release records existed (6D PR F, RT-27), with its date precision, source and reviewer, and without notifying anyone. Contract, fields and refusals: Importing past releases.

Preconditions

• Runbook 1 done (verified binding) and migrations 0012–0015 and 0021 applied (SYRF_MIGRATION_MODE=verify).
• The feature exists in the deployed image's feature definition catalog. Historical features without a definition (for example bulk PDF) need a feature-definitions.yaml change first.
• A reviewed input file per release: featureId, releaseKey, stage, precision, occurredOn, source, reviewer. Use Unknown when no date is evidenced; never guess one.
• No API outage or writer freeze is needed: the command writes no policy.

Steps

1. R11.1 Preview with SYRF_RUNTIME_RELEASE_HISTORY_MODE=preview, ..._INPUT (the reviewed input) and ..._OUTPUT (a new path), plus the Mongo settings, RuntimeEnvironment and the three SYRF_MIGRATION_BINDING_EXPECTED_* values.
2. R11.2 Review the preview: record (status Active, imported: true, no applied), event (HistoricalImported, provenanceDetail), the precision and occurredOn, and previewDigest.
3. R11.3 Apply the unmodified preview with SYRF_RUNTIME_RELEASE_HISTORY_MODE=apply, ..._INPUT (the preview), a new ..._OUTPUT and SYRF_RUNTIME_RELEASE_HISTORY_ACTOR=<operator identity>.

Preflight checks and expected output

Check	Expected output
R11.1	Exit 0; Historical release import preview completed; preview file written; nothing in the database changed
R11.3	Exit 0; receipt with the stored record and event; runtimeFeatureReleaseOutbox and runtimeFeatureFlagAudit counts unchanged
Replay	Applying the same preview to a new output path exits 0 with the original receipt and writes nothing

Evidence to capture

• The reviewed input, preview and receipt files, unedited, and the reviewer's approval.

If a step is refused

Refusals exit 2 with Historical release import refused: <message>. Nothing is written.

Message starts with	Action
Unknown feature	Add the feature definition first, in its own PR
The reviewed import preview was modified	Preview again; never edit a preview
The reviewed import preview is stale	The binding changed (runbook 9) or the input no longer gives the plan; preview again
A release with this feature and release key already exists	The key is taken by another import or a prepared release; check the existing record, or use a new key for a genuinely separate release
Required release index is missing or incompatible	Apply migrations (runbook 1 preconditions); the command never creates indexes
Output already exists	Choose a new output path

An exit 1 is an unexpected failure: retry the identical preview with a new output path. This includes a failure to write the receipt file: the receipt is written after the database commit, so the import may already be committed. The retry is an exact replay, which writes nothing and returns the original receipt.

Known gaps and TO CONFIRM items

These were found while checking the runbooks against main. None is fixed by this docs-only change. They are tracked, with the same numbers, in #4403. Each needs a tooling PR or an owner decision before the staging rehearsal (entry criterion 5).

1. No chart-rendered way to run SYRF_MIGRATION_MODE=status or verify in a cluster (R1.1, R1.5).
2. PR previews do not run bind or up (only index-init), so a D1 preview rehearsal cannot apply 0009–0021 today.
3. The lineage import runs through the API chart's runtimePolicyLineage Job (#4794; Delivery through GitOps), but the initialise branch has no in-cluster delivery yet: it needs the API's rendered FeatureFlags settings (runbook 2).
4. The freeze mechanism for staging's auto-merged promotions is not defined (R2.4).
5. No approved way to clear a stale writer-registry entry younger than 24 hours (runbook 2).
6. No chart value for EnvironmentCoordinator:ClientSecret on Identity. Raw env: must not carry secrets (runbook 5).
7. Whether reviewed chart values should replace the raw env: runtime settings (Mode, Binding, ProductionPolicyMode, FleetInventoryMode, FleetExpectedReplicas, report, collector and coordinator sections) before production.
8. The continuous collector (R4.7, #4748) is built, defaults to off and is enabled nowhere. Enabling it in staging and production is an owner decision. Production also needs the Auth0 machine application (gap 9) and the OD-23 decision on the camarades-infrastructure change. The capture script still supplies no desired GitOps provenance, so promotion reports cannot be Complete yet. Until the collector is enabled in an environment, the manual loop in R3.4 is the only way to keep observations fresh.
9. The token request for production Auth0 machine applications (its audience parameter) is not documented.
10. No workflow calls the coordinator routes, and nothing builds its capability intersection or completion targets from a report.
11. Resolved by #4795: the API image's read-only SYRF_RUNTIME_POLICY_MODE=candidate writes the candidate manifest of the binary (R7.3). Its catalog digest is the pinned one, and its in-repo output is checked against the offline checker.
12. The reviewed recovery procedure before restarting a quarantined reader is not written (R9.5).
13. rebind is not built (R9.7).
14. Resolved by #4414 (owner decision D7=A, spec 6C.5): a Legacy host against an imported authority serves the deployed values, writes nothing and registers read-only, so runbook 10 is executable; the runtime-fleet legacy-return scenario rehearses it.
15. Resolved by owner decision D8=A (slice 6C.6): production uses Initialise (no legacy document) instead of the lineage import, so no legacy policy document has to be created. Its in-cluster delivery is still gap 3.

16. The E2E mock OIDC provider issues the collector machine token since #4409 and the report-scope token (environment:report:read) since #4414, which the runtime-fleet legacy-return and initialise-policy-change scenarios use to collect a compatibility report. It still has no mapping for the coordinator scope.

17. No chart template runs the historical release import command or keeps its artifacts (runbook 11). The lineage Job (gap 3) is the pattern to follow.

Not a gap since #4407: production enables and restores need Collector mode and a running collector. Static, the default, refuses them after any rollout. That is a configuration and operations prerequisite (runbook 6).