Runtime feature flags staging rehearsal plan and production prerequisites

This page holds two written plans from the remaining delivery plan:

• the staging rehearsal plan (6C.3), with entry and exit criteria for Chris's later gate;
• the production prerequisites list (6C.4), with an approval record template.

Every step points to a runbook step in Operate runtime feature flags (R<runbook>.<step>). The local rehearsal that comes first is in the isolated rehearsal checklist.

Roll-forward means the next reviewed forward change if a step fails or must be reversed. Deployments and data only move forward (ADR-023); a rollback is a planned replacement release.

Staging rehearsal plan (6C.3)

NOT EXECUTED. This is a plan. No step below has been run in staging, in a PR preview or in production. Staging execution needs Chris's explicit decision (owner gate, delivery plan §8).

Scope

• One control: disableMembership (D2). Statistics keys are not enrolled as runtime decisions.
• API and PM are promoted together (D4).
• The rehearsal proves the mechanism: bind, import, enrol, collect, change, stop, restore and the Legacy-return checks. It is evidence about the mechanism, never authorisation for production.
• The RT-69 part of OD-26 (guide page and What's New entry appearing at the moment of the switch) runs as the RT-69 rows below, on a release of the rehearsal control. It needs RuntimeFeatureReleases:Enabled and RuntimeFeatureReleases:OutboxDispatch:Enabled on in staging, which is a separate owner decision (delivery plan §8); until then those rows are recorded as outstanding.

Stage 0: throwaway PR preview (D1)

Before staging, the same sequence may run in a throwaway PR preview with its own database and no real users, then be torn down. Gaps that block it today:

• previews do not run bind or up, so migrations 0009–0021 are never applied there (runbook 1 preconditions);
• the lineage command has no in-cluster delivery mechanism (runbook 2);
• the collector and report clients require distinct provisioned Secrets and an Identity chart with their Secret references enabled (runbook 4).

Until those are resolved, Stage 0 is limited to what the isolated rehearsal covers.

Entry criteria for Chris's gate

Each criterion names how it is verified.

1. 6A and 6B are merged (#4407, server-time retirement, merged as 9174f9e9). Verify: PR states.
2. The isolated rehearsal (6C.2) transcript shows every row of the checklist as expected, or as an accepted gap. Verify: the stored evidence directory.
3. Staging's API and PM run the same main build, which contains migrations 0009–0021. Verify: /health/live versions and migrations.lock.json at that commit.
4. The pre-deploy migration Job can be enabled in staging (pre-deploy plan R1). Verify: the reviewed cluster-gitops change.
5. The known gaps that block a step are resolved or explicitly accepted by Chris: status/verify delivery, lineage delivery, Identity secret wiring, collector enablement (the loop is built: R4.7), writer freeze mechanism. All 16 items are tracked in #4403; items 14 and 15 wait on owner decisions D7 and D8 on the programme progress page. Verify: each item has a linked PR or a recorded decision.
6. A staging read-only Kubernetes identity for the collector exists. With the continuous collector (R4.7) it is the chart-rendered Role syrf-api-workload-collector; for the manual path, the precedent syrf-stats-preflight. Verify: the rendered Role (exactly get on the two Deployments, list on ReplicaSets and Pods).
7. No active deployment constraint and no other runtime-policy change is planned during the rehearsal window. Verify: report deploymentConstraint projection is null.
8. A rehearsal window and a named operator are recorded in the approval record. Verify: the record.

Steps

Step	Runbook ref	Owner	Approver	Evidence	Roll-forward
S1 Read the live fingerprint	R1.1	Operator	Chris	status output	Read-only; none needed
S2 Enable the migration Job and bind	R1.2–R1.4	Operator	Chris (cluster-gitops PR)	PR URL; bind and up logs with epoch	A failed PreSync keeps the previous release serving. A failed init container keeps the API's old pods, while PM (Recreate) is down until the fix lands. Fix forward with a new commit; never a manual Job (#4740: staging ran this step on 3 November through cluster-gitops#1863). A wrong binding cannot be replaced (rebind not built): stop and escalate
S3 Verify migrations	R1.5	Operator	n/a	verify output or report migrations: Ready	Forward migration fix in a new release
S4 Provision the report and collector clients	R4.1–R4.2, R7.3	Operator	Chris (secrets owner)	Client registration change (no secrets)	Revoke by removing the secret (Identity revokes the client)
S5 Run the collector for API and PM: one manual capture and receipt per service, then enable the continuous collector (below) seeded with those IDs	R4.3–R4.7	Operator	Chris (cluster-gitops PR)	Capture files and receipts; collector PR, started and published log lines	Replay the exact capture on exit 3; disable the collector (enabled: false)
S6 Review the baseline JSON	R2.1	Operator	Chris	Baseline file, approval, provenance	Edit and re-review before S7
S7 Lineage preview	R2.2–R2.3	Operator	Chris (reviews the file)	Preview file and digest	Discard; make a new preview
S8 Freeze and scale the API to zero	R2.4	Operator	Chris	Freeze reference, zero-pod proof	Restore replicas with a forward change if S9 is abandoned before apply
S9 Lineage apply	R2.5–R2.6	Operator	Chris	Receipt file, logs	No undo. Correct values with a reviewed policy revision (S13). A wrong lineage needs rebind (not built)
S10 Enrol API and PM with inventory and Collector mode	R3.1–R3.3, R6.1–R6.3	Operator	Chris (cluster-gitops PR)	PR URL, rendered env diff	Forward values fix. Never return the API to Legacy (it refuses the imported authority)
S11 Confirm the collector resumed publishing (published lines for both services) or run the manual loop (cycle under 30 s); post-enrolment checks; end the freeze	R3.4, R4.7	Operator	n/a	Collector log lines or loop receipts; authority-status, authority-evaluation, fleet responses (Unknown with a Collector… reason is expected until fresh observations land)	Runbook 9 cases
S12 Policy-change report and checker	R7.3	Operator	n/a	Report file, checker output	Fresh report
S13 Flip disableMembership on	R7.1–R7.6	Operator	Chris	Preview, receipt, fleet Active timing	Reviewed forward revision (off) or stop (S15)
S14 Flip it off	R7.1–R7.6	Operator	Chris	As S13	Reviewed forward revision
S15 Emergency stop	R8.1–R8.3	Operator	Chris	Stop receipt, per-host adoption	Restore (S16)
S16 Restore	R8.4–R8.5	Operator	Chris	Preview, receipt, fleet Active	Another stop
S17 Lost-response drill	R7.7, R9.1	Operator	n/a	Lookup result; exact replay returns the original receipt	None needed
S18 Rollout drill: one ordinary API and PM rollout	R6 notes	Operator	Chris	Fleet states during rollout; old rows Retired; Active again	Forward change
S19 Legacy-return checks only (no mode switch)	R10.1–R10.3	Operator	n/a	Report legacyParity, checker output	Read-only. R10.4 is executable since #4414 but is not part of this rehearsal
S20 Optional: coordinator record, extend, release	R5.1–R5.4	Operator	Chris	Receipts, report before and after	Release, or break-glass (S21)
S21 Optional: break-glass drill	R9.6	Operator (human administrator)	Chris	Reason, receipt, report	None needed
S22 Store the evidence and approval record	All	Operator	Chris	Run record	n/a

Staging enablement of the continuous collector

NOT EXECUTED. Written plan only. Nothing here has been enabled in staging. It needs Chris's approval of the cluster-gitops PR.

The loop, its chart and an isolated proof are in #4748. The proof ran against the local runtime-fleet stack with a stand-in Kubernetes API, not a cluster.

1. Secret and Identity (S4): the namespace Secret exists (default syrf-environment-collector, key clientSecret, at least 32 characters). Identity runs with environmentCollector.enabled: true pointing at it.
2. API collector settings (R4.1): WorkloadCollector Enabled, Provider: OpenIddict, ClientId and Audience, and the binding values.
3. First publication by hand (R4.3–R4.5) with a reviewed context, recording each service's last accepted observation ID.
4. One cluster-gitops PR to the staging API values with the runtimeWorkloadCollector block from R4.7: apiOrigin, tokenUrl (<identity-origin>/connect/token), environment, bindingEpoch, expectedNamespace: staging and seedObservationIds from step 3. Review that the render is exactly the R4.7 resource list.
5. After sync: the log shows started (with the seeds as chain heads) and then published for both services about every 10 s. The report shows workloads:api and workloads:project-management Current.
6. Rollout check: at the next ordinary API or PM rollout, a new observation lists the new pods. published lines resume within 30 s of the rollout finishing.
7. Roll forward: runtimeWorkloadCollector.enabled: false stops it. Re-enabling needs fresh seeds (R4.7).

RT-69 release guide and What's New check (6G)

Accepted plan RT-68 and RT-69 (OD-16, OD-26): switching a release on makes its guide page and What's New entry available to its audience at the same moment; a stop hides the entry (the guide link stays, the in-app chip shows "Currently unavailable" once 6G.5 E2 ships); a restore brings it back with its original date; reaching general release publishes it on the public What's New dated that day. Each row stores captured responses and screenshots, before and after the step, with the release record.

Captured responses, every row:

• GET /api/projects/{projectId}/features/availability as a project administrator, and GET /api/features/availability (no project) as the same person and as an ordinary member: the feature's availability, release, announcement and whatsNew.
• GET /api/whats-new/public anonymously (no cookie, no token; curl -sS -D -), keeping the headers: Cache-Control: public, max-age=60 and the body. It is the same for every caller and carries published content only (D26 A).

Screenshots, every row: the home page signed out, the home page and the project overview ("New in this project") signed in as the administrator, light and dark. Staging delivers announcements exactly as production does (D60 A, 26 Oct 2026), so a delivered entry shows "New to you" and nothing is labelled "Rehearsal"; the staging binding keeps its stored testSink value, which is no longer read, so no re-bind is needed.

Step	Runbook ref	Owner	Approver	Evidence and expected result	Roll-forward
RT69-1 Before: release prepared, not applied	6D prepare	Operator	Chris	Availability: no release, whatsNew: null. Public: entries: []. Home page signed out identical to before 6G	None needed
RT69-2 Apply as LimitedRelease and wait for confirmation and the MaterialiseWhatsNew row Done	6D apply, R7	Operator	Chris	Administrator in the project: whatsNew present, dated by the LimitedRelease milestone; ordinary member and no-project answers: no entry. Public: still entries: [] (a limited release is never public). Guide page reachable from "Read the guide"	Emergency stop (S15)
RT69-3 Advance to GeneralRelease and wait for confirmation and its row Done	6D apply, R7	Operator	Chris	Public: one entry, releasedAtUtc equal to the GeneralRelease milestone, announcementId {releaseId}:GeneralRelease equal to the signed-in answer's announcement.id. Signed out the home page shows it; signed in it shows once	Emergency stop (S15)
RT69-4 Emergency stop	R8.1–R8.3	Operator	Chris	Within 60 s plus the 10 s release read: the administrator's answer is unavailable/stopped with whatsNew: null, and a fresh public request (no cache) has no entry. A shared cache may serve the old public body for up to 60 s; record the time it cleared	Restore (RT69-5)
RT69-5 Restore	R8.4–R8.5	Operator	Chris	The public body is byte-identical to RT69-3's (same date); the signed-in entry returns with the same releasedAtUtc; no "New to you"	Another stop
RT69-6 Switch-off check: RuntimeFeatureReleases:Enabled=false	Values change	Operator	Chris (cluster-gitops PR)	Both routes answer 404 {"code":"feature_availability_disabled"}; the home page signed out and signed in is the pre-6G page	Switch back on (values change)

Exit criteria

1. Every executed step has its evidence stored with the run record.
2. Each flip (S13, S14) reaches fleet Active on its committed identity within the 60-second confirmation window, with per-host adoption times recorded.
3. The stop (S15) reaches every API decision and every host acknowledgement; restore (S16) brings back the saved override.
4. A policy-change report is Complete in Collector mode, and the checker passes it (S12).
5. The rollout drill (S18) returns to Active without operator action.
6. The Legacy-return checks (S19) pass.
7. No step needed a production credential.
8. RT-69 (OD-26): every RT-69 row has its captured responses and screenshots and shows its expected result, or is recorded as outstanding while the release switches stay off in staging.
9. Chris reviews the record and decides on production separately.

Production prerequisites (6C.4)

NOT EXECUTED. This is a list of prerequisites, not a record of anything done. Every production step needs Chris's approval at the time; merging a production cluster-gitops PR deploys it.

Step	Runbook ref	Owner	Approver	Evidence	Roll-forward
P1 Catch-up promotion: production on main's runtime-flag code in Legacy mode, API and PM at the same build (D4)	CD plan, D4	Release operator	Chris (production PR)	Promotion PR, ArgoCD sync, /health/live versions for both services	A planned replacement release. The planned rollback to 104eca9 with migration 4's Down is the one built exception (ADR-023)
P2 ADR-023 migrations enabled in production, with the binding in the same change. First, the API and PM builds promoted to production must contain the #4740 change, so that the chart renders the migrate-bind/migrate-up init containers and the runner has the no-op fast path (ADR-023 amendment 2026-11-04). Without it, the values-only change lands as a hookless self-heal sync and production repeats #4740. NOT RUN: production has not enabled migrations, and doing so needs Chris's approval of the specific PR	R1.1–R1.5	Release operator	Chris (production PR)	The rendered Deployment showing initContainers; status output; bind and up logs (Jobs or init containers); verify output	A failed PreSync Job keeps the previous release. A failed init container keeps the API's old pods (RollingUpdate), but PM (Recreate) is down until the fix lands. Fix forward with a cluster-gitops commit, never a manual Job. A wrong binding needs rebind (not built)
P3 OD-21 automation identity: Auth0 machine application, scope environment:report:read, credential in a main-only GitHub environment	R7.3	Release operator	Chris (production-shared Auth0 tenant change)	Tenant change record (no secrets), API CompatibilityReport settings PR	Remove the grant or the application
P4 OD-23 collector: an Auth0 machine application for environment:observations:write (production-shared tenant change) and its client secret as a namespace Secret; the continuous collector (R4.7) enabled in the production API values with tokenUrl/audience for Auth0 (TO CONFIRM, gap 9). Owner decision: whether the camarades-infrastructure change OD-23 named is still needed now that the chart renders the read-only Role	R4.1–R4.7	Infrastructure operator	Chris (Auth0 change, production PR; camarades-infrastructure PR if kept)	Tenant change record (no secrets), production PR, first capture and receipt per service, collector published lines	runtimeWorkloadCollector.enabled: false, or disable WorkloadCollector
P5 Coordinator identity and workflow, scope environment:deployments:coordinate	R5.1–R5.4	Release operator	Chris	Workflow PR, Auth0 change, first receipts	Disable DeploymentCoordinator; break-glass for a stranded constraint
P6 Chart and configuration values for API and PM: Mode=Shared, ProductionPolicyMode=Runtime, SharedReaderMode=Disabled, binding, runtimePolicyHostMetadata.enabled, hpa.enabled: false, fixed replicaCount; API also FleetInventoryMode=Collector and both FleetExpectedReplicas	R3.1, R6.1–R6.4	Release operator	Chris (production PR)	Rendered env diff, post-rollout checks	Forward values fix, or ProductionPolicyMode=Legacy (runbook 10, once P11 is in the production build)
P7 D4 together-deploy and the #4354 mixed-version window	D4	Release operator	Chris	Both services on one build before any admin step	Finish the rollout; retry after Retry-After
P8 Collector-mode requirement: production admission follows FleetInventoryMode (since #4407), so set Collector and keep the collector publishing (P4)	R6.3, R3.4	Release operator	Chris	A production build containing #4407; a rollout drill showing old rows Retired after the 35-second quiet period (up to 15 minutes for rows from pre-#4407 images) and the fleet Active again	In Static mode (the default) enables and restores are refused after any rollout; stops still work
P9 Initialise the production authority from the deployed values, with the writer freeze (API outage). Production has no legacy policy document (legacy mutation is disabled there), so it uses initialise, not the lineage import (D8=A)	R2.I1–R2.I7	Release operator	Chris (window, reconciled rendered values, preview)	Input, both previews and the cmp result, reconciled API and PM rendered values, freeze reference, receipt, verify output	No undo. Correct values with a reviewed forward policy revision; a wrong lineage needs rebind (not built). Needs an in-cluster delivery mechanism first (gap 3). Rehearsed only on an isolated stack (run-local.sh --runtime-fleet-initialise, checklist)
P10 Recovery exercise and freshness trial (increment C)	Runbook 9, 6B evidence	Release operator	Chris	Exercise record; measured adoption times	n/a
P11 Legacy-return path in the production build: #4414 (D7=A) promoted to production before any import there	Runbook 10	Developer	Chris	Merged PR, the runtime-fleet legacy-return scenario evidence, and the production build containing it	n/a
P12 First enable as a release (OD-26): guidance under OD-5 and the RT-69 pre-enable check	R7, RT-69	Product owner	Chris	Guide page and What's New check before and after: the RT-69 rows with production's captured availability and public (GET /api/whats-new/public) responses and screenshots	Emergency stop (runbook 8)
P12a Release guidance bundle in the production API image (6D.5, D-Q6; D38 A, 6G.7): deploy an API image built since 6G PR G (#4513), which carries the server-<sha256>.json compiled from the same commit at /app/release-guidance/; leave RuntimeFeatureReleases:GuidanceBundlePath unset. No Secret and no chart value. Without such an image every guidance-required Apply is refused with GuidanceBundleMissing (emergency stops are unaffected)	Applying a release	Release operator	Chris (production promotion PR)	The image build's "Check release guidance bundles in the image" notice (bundle name and SHA-256), the promoted API image tag, an Apply preview showing guidanceState: Verified with the expected guidanceBundleDigest	Promote the previous API image (admission then verifies that image's bundle)
P13 Production enable of disableMembership	R7.1–R7.6	Release operator	Chris	Report, checker, preview, receipt, fleet Active	Emergency stop, then a reviewed forward revision
P14 Approval record completed for each of P1–P13 (including P12a)	Template below	Release operator	Chris	Stored record	n/a

Approval record template

Copy this block once per step or per rehearsal run. Use placeholders for anything secret.

Approval record
---------------
Step ID(s):                <S… or P…>
Environment:               <env> (binding epoch <binding-epoch>)
Runbook steps:             <R…>
Operator:                  <name>
Approver:                  <name>
Approval reference:        <chat message, PR approval or issue comment URL>
Window (UTC):              <start> to <end>
Freeze reference:          <reference, if runbook 2>
Builds:                    api <image@sha256>, project-management <image@sha256>
Preconditions checked:     <list, with the evidence for each>
Commands or requests run:  <list; no tokens, no secrets>
Expected output seen:      <yes/no per preflight check, with excerpts>
Refusals and handling:     <code, action taken, retry mutation IDs>
Evidence stored at:        <path or link>
Roll-forward needed:       <none / description and PR>
Outcome:                   <completed / stopped at step …>
Follow-ups:                <issue links>