Skip to content

Runtime flag-state metrics and the drift dashboard

Each API copy can report which runtime flag state it holds, through the metrics path the API already has (/metrics on port 9464, scraped by Google Managed Prometheus). A Grafana dashboard then shows where that state differs: between environments (a pilot on in staging and off in production, which is expected and should be visible), and inside one environment (a copy on an older policy revision, or one whose policy has not refreshed lately). This is the flag-state export D24 proposed; the owner approved it on 3 November 2026 (delivery plan, D24).

It is off by default everywhere. Turning it on is an environment step in cluster-gitops, written down below and not yet run. Alerting rules are a possible follow-up, not part of this.

How it works

flowchart LR
    P["API copy<br/>in-memory policy snapshot"] -->|gauges, no I/O| E[":9464 /metrics"]
    E -->|scrape every 60s| C["GMP collector"]
    C --> M["Cloud Monitoring<br/>project camarades-net"]
    M -->|PromQL| G["ARRNC Grafana<br/>runtime-flags dashboard"]
  • Meter. SyRF.RuntimeFlags, in src/libs/mongo/SyRF.Mongo.Common/RuntimeFeatureFlags/RuntimeFlagStateMetrics.cs. Its observable gauges read what the copy already holds in memory: the IFeatureDecisions captures, the bound authority reader's published snapshot (in Mode=Shared), or the runtime provider's last applied snapshot (in Legacy and Shadow modes). Reading them never touches MongoDB.
  • Setting. RuntimeFlagMetrics:Enabled (SYRF__RuntimeFlagMetrics__Enabled), from the chart value runtimeFlagMetrics.enabled (src/charts/syrf-common/env-mapping.yaml, default false). Off, the meter is never created. It is deployment configuration, not a feature flag: it changes no request handling.
  • Scrape path. The gauges reach GMP only while prometheusMetrics.enabled also serves /metrics and renders the PodMonitoring (Query BFF auth metrics describes that path). The default keepMetricsRegex (syrf_.*) already keeps these series. Staging and production both have prometheusMetrics.enabled: true.
  • Services. The API only. Project Management reads runtime flags too, but has no scrape endpoint yet; the meter lives in the shared library so adding PM is a scrape endpoint plus one registration (follow-up).

Series and labels

Default Prometheus translation, pinned by RuntimeFlagMetricsScrapeTests:

Series Per Value
syrf_runtime_flag_enabled flag, service 1 when the flag's global (context-free) decision on this copy is on, else 0. Not per project or person.
syrf_runtime_flag_targeted flag, service 1 when this copy would apply stored targeting for the flag: a rule, a rollout, or a consent setting. Always 0 outside Mode=Shared and in production.
syrf_runtime_flag_stopped flag, service 1 while an emergency stop is in force in the policy this copy holds (production counts only production-eligible keys' stops). Always 0 outside Mode=Shared: stops exist only in the bound authority.
syrf_runtime_policy_revision service The policy revision this copy last confirmed: the authority's revision in Mode=Shared, the legacy override document's revision otherwise. Absent until the copy has confirmed one.
syrf_runtime_policy_age_seconds service Seconds since this copy last confirmed its policy. While it never has, it counts from the copy's start-up, so such a copy still shows in panel 3.
syrf_runtime_shared_reader_age_seconds service Seconds since the legacy-format shared reader last read the policy document. Present only while that reader decides keys on this copy (RuntimeFeatureFlags:SharedReaderMode=Membership on the API, as in staging); absent otherwise.
  • Labels. flag (a key of the generated catalogue, nothing else) and service (api). Never a project, person or any other identifier; RuntimeFlagStateMetricsTests and RuntimeFlagMetricsScrapeTests scan every series against that allow-list. The exporter adds otel_scope_name="SyRF.RuntimeFlags". GMP adds project_id, location, cluster, namespace, job and instance (the pod); the app never sets them.
  • What "confirmed" means. In Mode=Shared the authority reader confirms on every successful read (every 8 to 12 s); a failed, stale or quarantined read keeps the last verified revision and its age keeps growing. Outside Shared mode the runtime provider confirms each time it applies a snapshot (every 30 s); a snapshot older than the revision it holds confirms nothing.
  • The shared reader has its own age. With SharedReaderMode=Membership the API decides disableMembership from the legacy-format shared reader (refreshed every 15 s), not from the runtime provider. The two read the same document separately, so the provider can look fresh while the shared reader is stale and disableMembership has fallen to its safe value (off). syrf_runtime_shared_reader_age_seconds shows that case; panel 3 lists it beside the policy age. A successful read resets it; while the document carries preserved authority, each refresh confirms it without a read.
  • Consistency is per family. Each gauge family is observed from its own capture, so one family's flags never mix two revisions of a policy source. Two families in one scrape can: syrf_runtime_policy_revision can say N while syrf_runtime_flag_enabled already shows N+1. Compare families (and copies) across scrapes, not inside one; at a 60 s scrape interval such a pair is rare and lasts one scrape.
  • Cardinality. Bounded by the registry: 3 series per catalogued flag (64 today) plus 2, so 194 series per API copy (195 with the shared reader's age), the same at every scrape. At the 60 s scrape interval that is as many samples a minute per pod: 194 (195 with the shared reader's age).

Enable it for an environment (not run)

Owner approval is needed per environment, and production stays off until the owner turns it on. None of these steps has been run.

  1. Staging. In cluster-gitops, syrf/environments/staging/api/values.yaml:
runtimeFlagMetrics:
  enabled: true

After the Argo sync, check read-only that the variable reached the pod, then query GMP:

kubectl -n syrf-staging get pod -l app=api-staging-syrf-api \
  -o jsonpath='{.items[0].spec.containers[0].env[?(@.name=="SYRF__RuntimeFlagMetrics__Enabled")].value}'
curl -fsS -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  'https://monitoring.googleapis.com/v1/projects/camarades-net/location/global/prometheus/api/v1/query' \
  --data-urlencode 'query=count by (namespace) (syrf_runtime_policy_age_seconds)'
  1. Production, with the owner's approval. The same block in syrf/environments/production/api/values.yaml, in its own reviewed cluster-gitops PR. The image must carry this change first.
  2. Import the dashboard into the ARRNC Grafana (below).

To turn it off, set enabled: false (or remove the block); the series stop at the next roll-out.

The dashboard

docs/observability/dashboards/runtime-flags.json (uid syrf-runtime-flags). Every query is PromQL against the Google Cloud Monitoring data source (query type PromQL, project camarades-net). Variables: datasource (default uid gmp-camarades-net), env_a and env_b (the two namespaces panel 1 compares, default syrf-staging and syrf-production), namespace (panels 2 to 5), service (api) and age_threshold (default 60 s, the fleet's 60-second follow target).

  1. Flags that differ between $env_a and $env_b (table: flag, one column per environment, targeted). A flag is listed when its global state differs, or when only one environment reports it (a newer image adds a flag):
max by (flag, namespace) (syrf_runtime_flag_enabled{service="$service", namespace=~"$env_a|$env_b"})
and on (flag) (
    (max by (flag) (max by (flag, namespace) (syrf_runtime_flag_enabled{service="$service", namespace=~"$env_a|$env_b"}))
      != on (flag)
     min by (flag) (max by (flag, namespace) (syrf_runtime_flag_enabled{service="$service", namespace=~"$env_a|$env_b"})))
  or on (flag)
    (count by (flag) (max by (flag, namespace) (syrf_runtime_flag_enabled{service="$service", namespace=~"$env_a|$env_b"})) < 2)
)

The targeted column is the same filter applied to label_replace(max by (flag) (syrf_runtime_flag_targeted{...}), "namespace", "targeted", "", ""). The table pivots with Grafana's Grouping to matrix (rows flag, columns namespace). 2. Copies behind their environment's highest revision (revisions behind; empty is healthy). Each service is compared only with its own copies, so API and PM copies are never measured against each other:

(max by (namespace, service) (syrf_runtime_policy_revision{namespace=~"$namespace", service=~"$service"})
  - on (namespace, service) group_right
 max by (namespace, service, instance) (syrf_runtime_policy_revision{namespace=~"$namespace", service=~"$service"})) > 0
  1. Copies whose policy is older than the threshold, in two queries: the policy age (A) and the shared reader's age (B):
max by (namespace, service, instance) (syrf_runtime_policy_age_seconds{namespace=~"$namespace", service=~"$service"}) > $age_threshold
max by (namespace, service, instance) (syrf_runtime_shared_reader_age_seconds{namespace=~"$namespace", service=~"$service"}) > $age_threshold
  1. Active emergency stops:
max by (namespace, service, flag) (syrf_runtime_flag_stopped{namespace=~"$namespace", service=~"$service"}) > 0
  1. Copies reporting (the positive control for 2 to 4: an empty drift panel means something only while its environment's copies report here):
count by (namespace, service) (syrf_runtime_policy_age_seconds{namespace=~"$namespace", service=~"$service"})

Every query passes promtool check rules (Prometheus 3.5.0) with the variables at their defaults, and evaluates on a real Prometheus in the rehearsal below. The panels' Grafana rendering (in particular panel 1's pivot) has not been checked in a Grafana instance yet; check it during the import.

Import it into the ARRNC Grafana (not run)

  1. Make sure the GMP camarades-net data source exists (uid gmp-camarades-net, type stackdriver; set-up in Query BFF auth metrics).
  2. In Grafana: Dashboards → New → Import, upload runtime-flags.json, keep the uid syrf-runtime-flags, and pick that data source for datasource.
  3. With staging enabled, open the dashboard: panel 5 should list syrf-staging api with the API's replica count. Check that panel 1 pivots to one row per flag; if Grafana names the value field other than Value, set it in the panel's Grouping to matrix transformation and re-export the JSON into this repository.

Isolated rehearsal

The rehearsal that proves panel 2 sees drift ran on Bramble's local runtime-fleet E2E stack (the 6B harness, two API and two PM copies in Mode=Shared), never against staging or production. Its scripts live outside the repository, in ~/workspace/syrf/handover/flag-metrics-evidence/ on Juniper:

  • run-rehearsal.sh <worktree> ships the PR head to Bramble as a git bundle, makes a scratch worktree there, runs rehearse.sh, copies the evidence back and removes the worktree.
  • rehearse.sh brings the stack up with bash e2e/run-local.sh --runtime-fleet --iterative, restarts the four copies through runtime-fleet-harness.sh --host-control with RuntimeFlagMetrics and PrometheusMetrics on (each API on its own metrics port, api-2 reaching MongoDB only through a TCP proxy), starts a local Prometheus scraping both APIs, runs rehearse.mjs, then tears everything down with bash e2e/scripts/teardown.sh.
  • rehearse.mjs freezes the proxy so api-2 holds the old revision while still answering scrapes, flips disableMembership on api-1, and checks that panel 2 (and, after 60 s, panel 3) lists api-2 and only api-2, that both drop out once the proxy is thawed and api-2 catches up, that a stop and a restore move syrf_runtime_flag_stopped (panel 4), and that every dashboard query evaluates.
  • validate-dashboard.sh <worktree> is the promtool check, with a deliberately broken expression as its control.

Result, 3 November 2026 (PR head 6d5aee5d9, before the shared reader's age and the per-service grouping of panel 2 were added; the rehearsal stack runs Mode=Shared with the shared reader disabled, so neither changes what it showed; evidence evidence-20261103T184141Z/ beside the scripts): 23 of 23 checks passed. Both copies reported (panel 5: 2) with 64 series in each per-flag family; panel 2 was empty before the hold. With api-2 held, panel 2 listed api-2 alone, one revision behind, about a second after the flip on api-1; panel 1 (api-1 and api-2 standing in for two environments) listed disableMembership; panel 3 listed api-2 alone once its age passed 60 s. Two seconds after the thaw api-2 was on the new revision and panels 1 to 3 were empty again. The stop showed syrf_runtime_flag_stopped = 1 on both copies (panel 4) with the value off, and the restore cleared it. The scrape carried only flag, service and otel_scope_name labels, and /metrics on the application port answered 404. The stack was torn down with bash e2e/scripts/teardown.sh.

Follow-ups

  • Project Management. Add a scrape endpoint to PM (move PrometheusScrapeEndpoint to a shared library, a chart port and PodMonitoring for PM), register the meter with service="project-management", and give PM's legacy path an IRuntimePolicyConfirmation (its shared reader's snapshot).
  • Alerting. Panels 2 and 3 translate directly into alert rules, if wanted.