MongoDB Database Reference¶

This document provides detailed reference information about SyRF's MongoDB database architecture.

Project-load concurrency¶

For member project loads, ProjectController.GetProject records project history on pmInvestigator and access on pmInvestigatorUsage. Each investigator save attempt loads an uncached aggregate inside the retry delegate. Repository caches are singletons and hold mutable objects, so writes must not reuse those objects across requests or retries. The usage delegate already reads directly from MongoDB through RetrieveOrCreate on each attempt.

MongoExtensions.SaveAsync filters by ID and the original version, then replaces with upsert enabled. A competing update can make that filter miss and produce E11000 on the existing ID. OnSaving increments the in-memory version before the write, even when it fails; retrying that same object can eventually match a newer version and overwrite the competing changes. Reloading and reapplying the operation preserves those changes. The unit of work also invalidates the repository cache on both successful and failed saves.

The project-load regression tests reproduce controlled contention against disposable local MongoDB, including preservation of another project view, a profile update, and a usage favourite. This demonstrates the failure mechanism; it does not establish the precise competing requests behind a staging incident. The existing three-retry limit and project authorization remain in place. This correctness fix is unflagged because it restores the intended concurrency protection without a new user workflow.

Database Instances¶

Atlas Clusters¶

SyRF uses two MongoDB Atlas clusters:

Cluster	Environments	MCP Server (for querying)
Prod cluster	Production	`mcp__mongodb-syrf-prod__*`
Preview cluster	Staging + PR previews	`mcp__mongodb-syrf-preview__*`

A local standalone container (mcp__mongodb-syrf-local__*) holds a backup snapshot of the prod cluster taken 2026-02-21. Useful for safe exploration without touching live data.

Environment Connections (Isolated)¶

Environment	Database	Cluster
Production	`syrftest` ⚠️	Prod
Staging	`syrf_staging`	Preview
PR Preview `{n}`	`syrf_pr_{n}`	Preview
Local development	Configurable	Local or local container

Warning: The production database is named syrftest despite its name suggesting a test environment. Treat it with care at all times.

How the charts bind a service to a cluster (no default)¶

The api, project-management and quartz Helm charts have no default cluster or database: mongoDb.clusterAddress and mongoDb.databaseName default to "", and each environment's cluster-gitops values must set both (production and staging in <env>.values.yaml; previews in preview.values.yaml plus the mongoDb.databaseName=syrf_pr_{n} ApplicationSet parameter), or name a ConfigMap through mongoDb.configMapName. If either is unset, empty, whitespace-only or the literal "", the render fails, for example:

syrf-common: mongoDb.clusterAddress must be set by the environment values (or set mongoDb.configMapName); refusing to fall back to a default

The ApplicationSets use ignoreMissingValueFiles: true, so before this guard (issue #4231) a missing environment values file silently bound a service to the production cluster the charts used to default to. The guard is generated from required: true on the two env vars in src/charts/syrf-common/env-mapping.yaml (helper syrf-common.requireEnvValue). E2E and the local stack do not render these charts; they set ConnectionStrings__MongoConnection__* on the processes directly. Identity's separate mongodb.* values are unaffected.

The same charts (plus identity and web) also require an explicit runtime environment: see Runtime Environment.

Legacy Databases (Prod Cluster)¶

Database Name	Purpose
`syrftest`	PRODUCTION database — all live data
`syrfdev`	Old development snapshot, mostly unused

Collection Architecture¶

Bounded Context Prefixes¶

Collections are named using a bounded context prefix derived from the entity's namespace:

// From MongoContext.cs
public static string GetBoundedContextCode(string? namespaceName)
    => namespaceName switch
    {
        null => throw new ArgumentException("AggregateRoot has no namespace"),
        _ when namespaceName.StartsWith("SyRF.ProjectManagement") => "pm",
        _ when namespaceName.StartsWith("SyRF.FileListings") => "as",
        _ when namespaceName.StartsWith("SyRF.LiteratureSearch") => "ls",
        _ => ""
    };

Collection naming formula: {prefix}{EntityClassName}

See MongoContext.GetCollection() at MongoContext.cs:154-163 for the implementation.

Project Management Collections (`pm` prefix)¶

The project management domain contains the core business entities:

Collection	Entity Class	Description
`pmProject`	`Project`	Systematic review projects with stages, memberships, questions
`pmStudy`	`Study`	Individual studies with screening, extraction, annotations
`pmInvestigator`	`Investigator`	User accounts and profile information
`pmSystematicSearch`	`SystematicSearch`	Literature search definitions linked to projects
`pmDataExportJob`	`DataExportJob`	Background export job tracking
`pmStudyPdfCorrection`	`StudyPdfCorrection`	PDF proposals and corrections (`Pending`, `PdfReplacedWithSuggestion`, `OriginalPdfKept`), written through `SubmitPdfCorrections` and decided through `ApprovePdfCorrections`. One index, `ProjectId_1_Status_1_StudyId_1` (pre-deploy migration `0035`, study management slice 9); nothing builds an index at startup. See below. The preview restore list used the wrong name `pmStudyCorrection` until #4807. The snapshot producer's list in cluster-gitops still does, so snapshots hold no PDF proposals yet (#4768)
`pmInvestigatorUsage`	`InvestigatorUsage`	User activity and usage statistics
`pmRiskOfBiasAiJob`	`RiskOfBiasAiJob`	AI-assisted risk of bias analysis jobs
`pmActivityClaimRevocationOutbox`	`ActivityClaimRevocationIntent`	Durable intent for the targeted claim-revoked SignalR event (#3720), written with the release under `reviewEligibilityPolicy`. Owner-only data; delivered and abandoned intents expire after 7 days (TTL on `DeleteAtUtc`)
`pmBulkStudyUpdateOperation`	`BulkStudyUpdateOperation`	ADR-020 operation record of an all-or-nothing Bulk Study Update: the attempt-scoped guard (lease, generation), phase, input manifest and counts. Permanent, value-free. Created on first use; dark until `bulkStudyUpdateAtomicApply`
`pmBulkStudyUpdatePlan`	`BulkStudyUpdatePlanEntry` (chunks of 100)	ADR-020 validated plan: each study's planned version, field changes and hashes. Deleted when the operation ends
`pmBulkStudyUpdateBeforeImage`	`BulkStudyUpdateBeforeImage`	ADR-020 before-image of each changed study (touched fields, content hashes, restore state). Expires 30 days after the operation is terminal (TTL on `ExpiresAtUtc`); a quarantined operation keeps its before-images
`pmRobRunOperation`	`RobRunOperation`	Application-authority M5b: the all-or-nothing save of one batch risk-of-bias run (`_id` = RoB job ID): lease, generation, phase, outcome, counts. `ActiveSearch` (unique partial index) is present while it applies or rolls back, so runs in one search are serialised. Permanent, value-free. Created on first use; dark until `batchRiskOfBiasAtomicApply`
`pmRobRunBeforeImage`	prior `RiskOfBiasInfoV2` per study	M5b before-image of each study a run wrote: its prior result exactly as stored (value, explicit null or absent) and restore state (`Applied`, `Restored`, `Superseded`). Expires 30 days after the run ends (TTL on `ExpiresAtUtc`)
`pmInboxNotification`	`InboxNotification`	The saved personal notification inbox (#3932, ADR-026): one row per recipient, kind and source occurrence (`RecipientId`, `Kind`, `ProjectId`, `SourceId`, `CreatedAtUtc`, `ReadAtUtc`, from N1 the additive `Source` sub-document, and from N2 the additive per-recipient `DismissedAtUtc`, `ResolvedAtUtc` and `Resolution`). Captured only while `notificationInbox` decides on and its indexes exist; flag off keeps every row, and the inbox API still serves people who have saved rows; a well-formed request from anyone else gets 404 (N2, M7). See below
`pmImportNotificationAdmission`	`ImportNotificationAdmission`	The import-completed notice's recipient, bound before the upload signature is returned (#3938, ADR-026): `_id` = the search ID, `ProjectId`, `RecipientId`, set once (`$setOnInsert`) and never changed. Written only while `notificationInbox` decides on and the inbox indexes exist. No secondary index (read by `_id` only). See below
`pmNotificationEmailPreferences`	`NotificationEmailPreferences`	One account's notification email settings (#3942, ADR-026 decision 13): `_id` = investigator ID, `Revision`, `Categories` (keyed by email category, each `Mode` and server-set `EnabledAtUtc`), `MutedProjectIds`, `AnyEnabled`, digest timing. Written only while `notificationEmail` (with `notificationInbox`) decides on and its indexes exist. See below
`pmNotificationEmailDelivery`	`NotificationEmailDelivery`	The email delivery ledger (#3942): one entry per saved inbox item (`_id` = the item's ID), `RecipientId`, `Kind`, `Mode`, `State` (`ready`, `leased`, `dispatching`, `sent`, `suppressed`, `ambiguous`), lease fields. Same admission as the preferences. See below
`pmNotificationEmailDigest`	`NotificationEmailDigest`	Daily digest windows (#3943, ADR-026 decision 16): one per recipient and local day (`_id` derived from both), with the frozen `NotificationIds`, the day's send time, `State` and `DueAtUtc`. Holds its members' ledger entries. Same admission as the preferences plus `0032`. See below
`pmNotificationFanOut`	`NotificationFanOut`	Recorded notification fan-out intents (N6, #4586, ADR-026 decision 14): one per occurrence too large to capture inline, `_id` derived from the kind and `SourceId`, with the row template, a recipient selector, lease fields and the expansion cursor. Written only by an inbox capture while `notificationInbox` decides on; expanded only while it decides on and its index exists. Never deleted. See below
`pmStudyConversation`	`StudyConversation`	Private one-to-one reconciliation conversations between a reconciler and one candidate reviewer (#3944, made private by #3965 in L7; ADR-026 decisions 17 and 22): `_id` derived from the reconciler's request and the candidate's session, `ProjectId`, `StudyId`, `StageId`, `ReconcilerId`, `Reviewer` (the candidate's session and investigator IDs and the label shown when it opened), `Exposure` (the questioned session, its status then, first question and first reply times), `Posts` (at most 100, 4,000 characters each; a candidate's post records their session status), `About` (optional, additive: the one form question the conversation is about, `{QuestionId}`; owner decision Q3), `ReadReceipts` (optional, additive: post ID in `N` format to the first time the other participant really opened it; written once by a pipeline `$mergeObjects` in which a stored time wins; owner decision Q13), `Revision`. A post also resolves its author's own notices of the thread in the same transaction (transition `replied`). Written only while `reconciliationConversations` (with `studyAttention` and `notificationInbox`) decides on and its index exists; each post's one notice (for the other party) goes to `pmInboxNotification` in the same transaction. Kept while the flag is off. Documents the L6 build saved (a `Reviewers` list, no `Reviewer`) are kept and never read. See below
`pmStudyPdfAvailability`	`StudyPdfAvailability`	Per-study PDF availability (study management slice 8): one document per study (`_id` = the study's id), `ProjectId`, `Status` (`Available`, `Unavailable`, `Unverified`, `External`, stored by name), `CheckedAt`, `CheckedLinkKey` (a fingerprint of the link the result is for) and `CheckStartedAt` (the claim lease). Read and written only while `studyPdfStatus` decides on and its index exists; never touches `pmStudy`. Kept while the flag is off. See below
`pmPublication`	`Publication`	[TARGET - Phase 12] Approved future system-scoped bibliographic identity, paired with immutable project-scoped `Citation` values. Migration, privacy and cross-project authorisation remain design gates — see three-level data model

Schema-v0 conditional parent answers¶

Schema-v0 annotation targets normally persist boolean and single-option conditions in legacy fields. Multi-option schema-v0 conditions use the additive ConditionalParentAnswers subdocument defined in ADR-011. Because project writes replace the full document, all API and project-management writers must be on that compatibility release before such data is created; older writers are below the safe rollback floor. The compatibility release is the permanent minimum writer version; the temporary runtime gate has been retired.

Reliable creation times (`createdAtUtc`)¶

pmProject and pmInvestigator documents created from spec 6F.F2 onwards carry createdAtUtc, a UTC BSON date (whole milliseconds) set once by the constructor that makes the new aggregate and never changed (Project.CreatedAtUtc, Investigator.CreatedAtUtc). It is additive and unflagged:

Existing documents are never backfilled or rewritten. A document without the element reads as null, and a save writes no element (the member is IgnoreIfNull), so a record created before the field keeps exactly its stored shape. No migration, index or start-up work exists for it.
Project.CreationDate is unchanged (the server's local clock, DateTime.Now); createdAtUtc is the reliable value.
Clock: the writing host's DateTime.UtcNow at construction. Aggregates are saved by a whole-document replace (ReplaceOne, upsert), which cannot take a server-side $$NOW, and the in-memory aggregate must carry the value or its next replace would drop it. Hosts keep NTP time, so the skew against the database clock is the cluster's clock synchronisation bound (well under a second).
Rollback: a binary that predates the field reads it into Entity.ExtraElements ([BsonExtraElements]; neither class map is strict) and writes it back on its next replace, so a rollback neither fails nor drops it (BsonClassMapTests, CreatedAtUtcPersistenceTests).
Readers: the project authority read (ProjectAuthorityFacts.CreatedAtUtc) and the Investigator authority snapshot (AuthoritySnapshot.CreatedAtUtc, a projected field) carry it to the feature target context, where its only reader is an exact rollout Pause: a subject created at or after the pause is kept out (Managing Feature Flags). It is never an authorization input.

Notification inbox (`pmInboxNotification`)¶

The saved personal inbox (ADR-026). The collection is new and empty until notificationInbox is switched on somewhere. Its indexes exist only as pre-deploy migrations in PmMigrations (ADR-023); InboxNotificationRepository builds none at start-up, and every consumer (export capture, the inbox API, InboxInvalidationWorker) asks IInboxAdmission first, which keeps the inbox closed while any of them is missing: no capture, 503 on reads, no change stream. From N2 the inbox API serves, with the indexes present, every caller while the flag is on and only callers with saved rows while it is off (a well-formed request from anyone else gets 404); with the indexes missing it answers 503 while on and 404 while off.

Index (migration)	Purpose
`RecipientId_1_CreatedAtUtc_-1__id_-1` (`0023`)	A recipient's list, newest first; serves the `(CreatedAtUtc, _id)` keyset cursor in order
`RecipientId_1_ReadAtUtc_1` (`0024`)	A recipient's unread count (`ReadAtUtc` null)
unique `RecipientId_1_Kind_1_SourceId_1` (`0025`)	One row per recipient, kind and source occurrence: a replayed capture converges on the existing row and keeps its read state; also finds the row a workflow resolves
`RecipientId_1_ReadAtUtc_1_DismissedAtUtc_1_ResolvedAtUtc_1` (`0026`, N2)	A recipient's unread-count candidates: `{RecipientId, ReadAtUtc: null, DismissedAtUtc: null, ResolvedAtUtc: null}`, equality on every key (absent fields index as null). Makes `0024` redundant (kept: the catalogue is append-only)

Writers. DataExportJobRepository upserts the export-completed row (_id = the export job ID) in the same transaction as the terminal export write, insert-only ($setOnInsert), only while the inbox is admitted. SourceInboxCapture (#3938) upserts the importCompleted row inside the import-completion transaction (Project Management) and the projectInvitation row inside the invitation save (API, known accounts only), keyed on the unique source identity (0025) with a new _id, insert-only. Each host decides once per operation, before the transaction, through IInboxAdmission. From N1 (C15 v2) every writer, the export and stage captures included, writes through the one capture path, InboxNotificationWriter, and each stores exactly the elements it stored before plus Source (below).
Source (N1, additive, nullable). A sub-document naming what the notice is about: Type (dataExportJob, systematicSearch, projectInvitation, stage), EntityId (the job, search, invitation or stage ID), ProjectId (null for a global kind, which also stores no top-level ProjectId), StageIds and TaskId (no element when unset). From N7 a feature release (Type featureRelease, EntityId the release ID, no project) also carries FeatureId, AnnouncementId ({releaseId}:{stage}) and AnnouncementRevision (no element on any other kind; an earlier build keeps them in Source's extra elements). It is set once on insert and never updated. Rows saved by L1-L3 have no Source and are never backfilled (no data migration); every read works from the typed fields, so they list exactly as before. A build that predates Source keeps it in the aggregate's extra elements (rollback-safe), and Source itself keeps elements a newer build adds. No index covers it.
Import admission (pmImportNotificationAdmission). SearchController binds the uploader as the recipient (keyed by search ID) before it returns the upload signature, only while the inbox is admitted; a different principal or project for the same search ID gets 409. Completion reads the binding inside its transaction and never guesses a recipient when there is none (legacy, living-search or system imports, or uploads made while off). A binding whose project differs from the completing import's, or that has no recipient, is treated as none: no row, and a warning that names IDs only. A binding made while on whose import completes while off stays inert. Flag off writes and reads nothing here. Rollback leaves the rows; no earlier build reads them. InboxNotificationRepository.MarkReadAsync sets ReadAtUtc on the caller's own row.
Per-recipient state (N2, additive, nullable). DismissedAtUtc (set once by dismiss or clear all, removed with $unset by restore), ResolvedAtUtc and Resolution (Reason: decided, completed, withdrawn, expired, superseded; SourceTransition, for example accepted or declined; ResolvedByRef, an opaque actor reference that no API returns yet; extra elements kept). No element exists until set, and rows saved earlier are never backfilled. Dismiss, restore and clear all are recipient-filtered UpdateOne/UpdateMany calls on the caller's rows. Only a workflow resolves: InboxNotificationWriter.ResolveAsync updates the existing unresolved row matched on (RecipientId, Kind, SourceId) (0025) inside the source transaction, never upserting; today the invitation answer (projectInvitation, completed, accepted/declined) while the inbox is open. A build before N2 reads these fields into its extra elements and updates rows only field by field, so it keeps them.
Clear all, option B (additive, nullable; #4726). StoredAtUtc: when the row was first saved, by the saving host's clock, set $setOnInsert by every insert (InboxNotificationWriter, inline captures and the fan-out expander alike) and never changed; CreatedAtUtc stays the event's time, which a recorded fan-out can save later. DismissBatchId: the Clear all call that dismissed the row (a fresh ID per call), removed with $unset by every restore together with DismissedAtUtc. Neither is backfilled: a row without StoredAtUtc is one saved before it existed, and Clear all then goes by CreatedAtUtc. No index: Clear all's read is 0023's recipient and creation-time range. A build before option B keeps both in its extra elements.
Readers. InboxNotificationRepository (recipient-scoped list with a status filter, unread-count candidates, get), and InboxInvalidationWorker's change stream (one per API pod, only while admitted; it sends an empty InboxChanged hint to the recipient's group).
Flag off. Nothing captures, resolves or watches; rows are kept with their state. From N2 the inbox API still serves a recipient who has saved rows (one existence read on RecipientId, limit 1) and answers a well-formed request from anyone without any with 404 (M7).
Rollback. Down 0026 drops the N2 index (an N1 build does not use it); Down 0025–0023 drops the indexes their Up created; the rows stay (no earlier build reads them). Retention and account-deletion erasure are not built yet (follow-ups in the notifications plan).

Notification email (`pmNotificationEmailPreferences`, `pmNotificationEmailDelivery`)¶

Opt-in email for saved notifications (#3942, ADR-026 decision 13). Both collections are new and empty until notificationEmail is switched on somewhere. Their indexes exist only as pre-deploy migrations; neither repository builds one at start-up, and the preference API and NotificationEmailWorker ask INotificationEmailAdmission first (notificationEmail and notificationInbox from one decision capture, then the inbox index set 0023–0026 and migrations 0027–0030; the daily digest worker and the admission's check also need 0032 on pmNotificationEmailDigest, #3943, ADR-026 decision 16).

Index (migration)	Collection	Purpose
`Mode_1_State_1_AvailableAtUtc_1` (`0027`)	delivery	The immediate claim: due `ready` (or expired `leased`) entries, oldest first. Adopts the identical index an earlier build of #3942 created at start-up
`RecipientId_1_Kind_1_State_1` (`0028`)	delivery	Moving a recipient's `ready` entries of a category's kinds between immediate and daily
`State_1_LeaseExpiresAtUtc_1` (`0029`)	delivery	Recovering dispatches whose lease expired as `ambiguous`
`AnyEnabled_1__id_1` (`0030`)	preferences	The worker's keyset page over accounts with any category on
`State_1_DueAtUtc_1` (`0032`)	`pmNotificationEmailDigest`	The digest worker's window claim (`ready`, or `leased` past its lease) and recovery of expired dispatches

Writers. NotificationEmailPreferencesRepository.TrySaveAsync (versioned; keeps unknown category keys; sets EnabledAtUtc itself; recomputes AnyEnabled). The ledger is admitted insert-only by the worker's discovery over saved pmInboxNotification rows, leased and acknowledged by the worker; dispatch admission increments the preferences' DispatchFence in the same transaction so a concurrent opt-out, timing change or mute conflicts with it. NotificationEmailDigestRepository prepares one window per recipient and local day in a transaction that holds its members' ledger entries (daily, leased, owner digest:{windowId}, no LeaseExpiresAtUtc), so the immediate claim and recovery never touch them; the window carries the lease, and its dispatch admission fences the preferences too. Preparation also finishes the ready entries of the daily kinds that no digest will take as suppressed, with Reason leftOutOfDigest (notice read, dismissed or resolved) or tooOld, so no entry stays ready for ever.
Readers. The preference API (the caller's own document) and the worker.
Flag off. Nothing reads or writes either collection; every row is kept and used again when the flag is back on.
Paused (NotificationEmailDelivery:Paused, the default): discovery still admits ledger entries; nothing is claimed or sent, and no digest window is prepared.
Rollback. Down 0030–0027 drops the indexes their Up created and keeps an adopted one; the rows stay (no earlier build reads them). Down 0032 drops the window index; windows stay, and an earlier build never claims a held entry. Retention and account-deletion erasure (windows included) are follow-ups (E32).

Recorded notification fan-out (`pmNotificationFanOut`)¶

Recorded fan-out intents (N6, ADR-026 decision 14). When one occurrence has more recipients than its kind may capture inline (never more than 200), the source transaction writes one intent here instead of the pmInboxNotification rows, and NotificationFanOutExpander writes the rows later in leased, bounded batches. The collection is new and empty until notificationInbox is on and such an occurrence happens.

Index (migration)	Purpose
`State_1_DueAtUtc_1` (`0031`)	The expander's claim scan: pending intents by when they are next due

Document. _id = SHA-256 over inbox-fanout:v1|{Kind}|{SourceId} (recording the same occurrence again is an insert-only no-op); the row template (Kind, SourceId, ProjectId, Source, WritesStage, StageId, OccurredAtUtc); Selector (Type: frozenList with RecipientIds, projectCapability, or from N7 featureReleaseAudience, which needs no other field because the release is in Source; optional ProjectId, StageId, Capability); State (Pending, Completed, as strings); DueAtUtc (recording time, lease expiry while leased, or retry time); LeaseOwner, LeaseToken; Cursor (the selector's opaque position after the last committed batch); RowsWritten, Batches, Failures, LastErrorClass (an exception type, never a message); from N7 the additive SkippedRecipients and SkippedRecipientIds (the latest 100; no element until a selector skips a recipient whose availability stayed undecidable); RecordedAtUtc, CompletedAtUtc. Unknown elements are kept on read (C16 floor).
Writers. SourceInboxCapture.RecordStageFanOutAsync (API and Project Management) records inside the source transaction. From N7 the API's FeatureReleaseNoticeRecorder records a release's intent from its What's New outbox row, insert-only outside any transaction (_id from the announcement ID and revision), only while featureReleaseNotices and notificationInbox decide on and RuntimeFeatureReleases:Enabled. NotificationFanOutStore claims, advances and releases for the expander; every write after the claim is fenced on LeaseToken, and the cursor advance commits in the same transaction as the batch's rows.
Readers. Only the expander (API). An intent whose selector type the build does not know is never claimed.
Flag off. Nothing reads or writes the collection; intents keep their progress and resume when it is back on. With notificationInbox on but featureReleaseNotices off, a release intent is claimed and given back a minute later (deferred, no failure counted, cursor kept).
Rollback. Down 0031 drops the index; the intents stay (no earlier build reads them). Completed intents are kept; retention is a follow-up with the inbox's.

Reconciliation questions (`pmStudyConversation`)¶

Private one-to-one reconciliation conversations (#3944, notifications restart L6; one-to-one since #3965, L7; ADR-026 decisions 17 and 22). The collection is new and empty until reconciliationConversations (which needs studyAttention) is switched on somewhere. A conversation saved by the L6 build has a Reviewers list and no single Reviewer; the L7 build keeps it and skips it on every read (gap C7: it may show every post to several reviewers).

Index (migration)	Purpose
`ProjectId_1_StudyId_1_StageId_1_DateTimeCreated_-1__id_-1` (`0033`)	A study's conversations in one stage, newest first; the participant filter applies within those bounds. Its prefix also serves the questioned-marker read (candidate sessions in one study and stage, exposure fields only), so L7 adds no index

Writers. StudyConversationRepository (API): a question opens one thread per candidate, all in one majority transaction (all, an exact replay, or nothing); an append is one transaction with the post's notice for the thread's other party (through InboxNotificationWriter, keyed on the post; the conversation is the row's Source) and the resolution of the notices its author received in the thread (owner decision Q5). Stable request and post IDs make retries converge; a changed payload or stale revision writes nothing. Opening the conversation marks the caller's notices of the thread read (pmInboxNotification, by their deterministic _ids). A real open (the conversation page, a notice's card) and a reply record ReadReceipts once, without touching Revision; inbox housekeeping never writes them.
Readers. The conversation routes and the two notice resolvers (API), after IStudyConversationAdmission. The reconciler's read receipts (owner decision Q13) come from the thread itself (no extra read, no new index).
Flag off. Nothing reads or writes the collection; conversations and notices are kept and readable again when on.
Rollback. Down 0033 drops the index; the conversations stay (no earlier build reads them: the L6 build pins studyAttention to its deployed value, off everywhere). Retention follows the integration plan's audit-record decision (D3-25).

Per-study PDF availability (`pmStudyPdfAvailability`)¶

What SyRF last found when it checked one study's PDF (study management slice 8a, studyPdfStatus, default off). The result lives outside pmStudy so a check never bumps the study's Audit.Version, never takes the study write lock and can't be written back stale by a review save; it is allowed on a bulk-locked study for the same reason. The collection is new and empty until the flag is switched on somewhere.

Index (migration)	Purpose
`ProjectId_1_Status_1` (`0034`)	A project's PDF statuses, and the Library's PDF filter (slice 8b)

Writers. StudyPdfAvailabilityRepository (API), from the in-process check worker only: a conditional upsert claims the study (no CheckStartedAt, or one at least two minutes old; a fresh claim makes the upsert collide on _id, so only one pod checks), then the result is written only while that claim still holds and the claim is cleared.
Readers. The review and reconciliation retrievals and the Library and single-study reads (API), after IStudyPdfAvailabilityAdmission, by study id. A missing document, or one whose CheckedLinkKey no longer matches the study's link, reads as "Not yet confirmed".
Deleted studies. Reads join on the live study (ProjectId, StudyIdLoad.UnlessDeleted), so a document for a deleted, moved or erased study is ignored. Nothing removes it: a study restored after a search deletion keeps its status. Documents left behind go when the flag is retired.
Flag off. Nothing reads or writes the collection and every PdfStatus is null; documents are kept and count again when it is switched back on.
Rollback. Down 0034 drops the index; the documents stay (no earlier build reads them).

PDF proposals (`pmStudyPdfCorrection`)¶

Proposed PDFs for a study (StudyPdfCorrection): a web address, or an older uploaded file, with Status (Pending 0, PdfReplacedWithSuggestion 1, OriginalPdfKept 2, stored as an int), SubmittedByInvestigatorId, ResolvedByInvestigatorId and Audit (Created is when it was proposed). Every current build reads and writes it. Study management slice 9 (studiesWorkspace) lists and counts the waiting ones: ProjectId is the project, Status is Pending, there is no CheckedProposalId element (L9, #3947, binds a proposal to a checked upload with it), and the study is live in the project.

Index (migration)	Purpose
`ProjectId_1_Status_1_StudyId_1` (`0035`)	A project's waiting proposals: the Overview count, the Proposed PDF waiting filter's distinct studies, and the per-study proposals read

Writers. SubmitStudyCorrection (a whole-document save). With studiesWorkspace on, Use proposed PDF and Reject decide through StudyCorrectionRepository.TryDecideAsync: one conditional update (still waiting, in its project, on its study) that moves Audit.Version, so a stale whole-document save can't write "waiting" back. Use runs it in one transaction with the study's link compare-and-set. With the flag off, Approve and Reject save the whole document as before.
Readers. The Library's per-row resolver (by StudyId, unindexed, every state), and slice 9's reads after IStudyPdfProposalsAdmission (the flag, then this index). The slice 9 reads project the documents (BsonDocument), so a CheckedProposalId document is seen and left out.
Flag off, or the index missing. The slice 9 reads answer Available: false and read nothing; Use and Reject need no index.
Rollback. Down 0035 drops the index. The admission keeps a positive check for the life of the process, so restart the API after a Down (the rollback runbook's step); the reads then refuse safely.

Source code locations:

Domain models: src/libs/project-management/SyRF.ProjectManagement.Core/Model/
Repositories: src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/Repositories/

Runtime feature release collections (no prefix)¶

Runtime feature releases keep their data in the project-management database, written only by the API's SyRF.API.RuntimeAdministration module. Their indexes exist only as pre-deploy migrations in PmMigrations (ADR-023); nothing builds them at start-up, and each writer refuses to run without the indexes it needs. See Managing Feature Flags.

Collection	Indexes (migration)	Description
`runtimeFeatureRelease`	`{environmentId, status}` (`0012`); unique `{environmentId, featureId, releaseKey}` (`0021`)	One record per release ID, advanced by a version compare-and-set
`runtimeFeatureReleaseEvent`	unique `{mutationId, type}` (`0013`); `{releaseId, recordedAtUtc}` (`0014`); `{environmentId, featureId, recordedAtUtc}` (`0015`)	Append-only release events; the unique index makes each event exactly-once
`runtimeFeatureReleaseOutbox`	unique `{dedupKey}` (`0016`); `{state, nextAttemptAtUtc}` (`0017`, the dispatcher's due scan); `{releaseId, state}` (`0022`, a release's rows by state)	Announcement side effects queued by Apply. `Done`, `Cancelled` and `Failed` rows are kept permanently: no TTL index and no clean-up job
`runtimeFeatureReleaseTestSink`	`_id` only (the deduplication key)	Retired (D60 A, 26 Oct 2026): staging's stand-in for in-app delivery before D60, one row per notice. Nothing writes or reads it any more; existing rows stay until a later clean-up (staging only)
`runtimeFeatureDiscoveryState`	`_id` only (`"{userId:D}:{announcementId}"`); no migration	One person's seen and dismissed state for one release announcement (6E.3): `userId`, `announcementId`, `releaseId`, `revision`, `seenAtUtc`, `dismissedAtUtc`, `updatedAtUtc` (MongoDB time), no project field. Written by one upsert by `_id` from `PUT api/account/announcements/{id}/state` and read by `_id` `$in` from feature availability; only `RuntimeFeatureDiscoveryStore` names it (`RuntimeReleaseCollectionsForwardRuleTests`). Account deletion does not erase it yet: D27 option A, pending Chris, leaves erasure to the OD-6 subject-deletion work
`runtimeFeatureConsent`	`_id` only (`"{featureId}:{subjectType}:{subjectId:D}"`); no migration	One project's or person's consent choice for one beta feature (6F.7, D29 A): `featureId`, `subjectType` (`project` or `person`), `subjectId` (lower-case `D` string, never `BinData`), `environmentId`, `bindingEpoch`, `choice` (`Joined` or `Left`), `consentMode` (the effective consent when chosen), `policyRevision`, `actorId` (the identity the request ran as), `realActorId` (the signed-in person who made it; the administrator under support impersonation), `recordedUnderImpersonation`, `realActorRecordedAtUtc` (equal to `updatedAtUtc` unless an older build rewrote the row since; the three are additive and absent on older rows), `chosenAtUtc` and `updatedAtUtc` (MongoDB time); no names or emails. Written by one upsert by `_id` from `PUT api/account/features/{featureId}/consent` and `PUT api/projects/{projectId}/features/{featureId}/consent`; read by `_id` `$in` for the features a request decides with consent, and counted by one anchored `_id` range (`"{featureId}:"` to `"{featureId};"`) for `GET api/runtime-feature-flags/features/{featureId}/choices`. Reads ignore a row of another binding epoch (the next write replaces it). Only `RuntimeFeatureConsentReader` and `RuntimeFeatureConsentStore` name it (`RuntimeReleaseCollectionsForwardRuleTests`). Account or project deletion does not erase it yet: erasure joins the D27 / D34 subject-deletion follow-up
`runtimeHelpPublication`	`_id` only (`"{environmentId}:{topicId}:{variantId}:{revision}"`); no migration	The help publication record (6G.P): one row per environment and public help topic variant revision that the environment's own public guide site was seen to serve (an anonymous `GET` of the guide image's `/syrf-publication.json` manifest and of the page: `200`, no redirect, section anchor present). Set on insert only: `environmentId`, `bindingEpoch` (CSUUID), `clusterFingerprint` (the verified binding's), `featureId`, `topicId`, `variantId`, `revision`, `publication` (`PublicPreview` or `PublicReleased` only), `path`, `anchor`, `guideOrigin`, `firstObservedAtUtc` (MongoDB time), `firstManifestDigest`, `firstObservedBy` (the pod's instance UID); updated every pass: `lastObservedAtUtc`, `lastManifestDigest`, `lastPageSha256` (SHA-256 of the served HTML), `lastGuideOrigin` (the origin that served this observation; additive, older rows lack it). A page counts as served only for `200`, `text/html` and a body containing `<html`. No people, request data or IP addresses. Written by one unordered majority bulk upsert by `_id` per pass of the API's `RuntimeHelpPublicationRecorder` (Shared-mode writer, `RuntimeFeatureReleases:Enabled`); read by one anchored `_id` range per topic variant (`RuntimeHelpPublicationStore.FindAsync`, the contract 6H will use with `RuntimeHelpPublicationEvidence.IsProductionEvidence`). Kept indefinitely (it is the evidence; it grows only with page edits). Only `RuntimeHelpPublicationStore` names it (`RuntimeReleaseCollectionsForwardRuleTests`). See Managing Feature Flags

Which component needs which index. Each one checks its indexes before it writes, and refuses (writing nothing) when an index is missing or incompatible:

Component	Required indexes
Prepare	the audit indexes `0009`–`0010` on `runtimeFeatureFlagAudit`, plus `0012`–`0015` and `0021`
Apply	the audit indexes `0009`–`0010` on `runtimeFeatureFlagAudit`, plus `0012`–`0017` and `0021`
Reconciler (confirmation, stop and restore history)	`0009`–`0010`, `0012`–`0015`, `0021`, plus `0022` for the stop-time cancellation
Outbox dispatcher	`0009`, `0010`, `0013`, `0016`, `0017` and `0022`. It also refuses if the outbox has any TTL index
Historical import (operator command)	`0012`–`0015` and `0021`. It writes no outbox row

The availability reader (6E.2) reads runtimeFeatureRelease only: one bounded, projection-only query that uses no new index. The admin "Ready to settle" read (6G AD3) is read-only and also adds no index: the policy document by _id, then bounded reads through 0012 (released GeneralRelease records), 0014 (their stop and restore events) and the audit's 0010 (the current lineage's receipts, newest first, at most 500); see Ready to settle. Two leases are stored on the documents themselves: reconcileLease on a record (30 s), and lease with state: "Leased" on an outbox row (60 s). Both are taken and expire against MongoDB's $$NOW, so a crashed holder releases its work without a clean-up job. The retired test sink never had a migration (its first insert created it). Per-person discovery state (6E.3, #4460) is not on main yet.

Identity Collections¶

The Identity database contains IdentityInvestigatorMappings, whose one-row-per-InvestigatorId tombstones permanently record mapping history and survive account deletion, and IdentityEmailReservations, whose holder set serializes claims on canonical email addresses. These collections deliberately live beside IdentityUsers, not in the project-management database, so Identity accounts and their mapping state share one backup and restore surface.

Other Bounded Contexts¶

Prefix	Namespace	Example Collections
`as`	`SyRF.FileListings`	File attachment storage
`ls`	`SyRF.LiteratureSearch`	Search configuration
(none)	Other	Uses entity name directly

GUID Representation¶

CSUUID (C# Legacy) Format¶

All document IDs in SyRF use CSUUID (C# Legacy GUID) format, which stores GUIDs as BinData subtype 3.

Configuration (from MongoUtils.cs):

public static void EnsureLegacyGuidSerializer()
{
    try
    {
        BsonSerializer.RegisterSerializer(new GuidSerializer(GuidRepresentation.CSharpLegacy));
    }
    catch (BsonSerializationException e)
    {
        if (BsonSerializer.LookupSerializer<Guid>() is not GuidSerializer
            {
                GuidRepresentation: GuidRepresentation.CSharpLegacy
            })
        {
            throw new InvalidOperationException(
                $"The Guid serializer is not the expected representation...", e);
        }
    }
}

Why This Matters¶

Format	BinData Subtype	Byte Order	Use in SyRF
CSUUID (Legacy)	3	Little-endian first 3 groups	Yes - all IDs
Standard UUID	4	Big-endian throughout	No

The byte order difference means the same GUID string produces different binary representations:

GUID string: 550e8400-e29b-41d4-a716-446655440000

CSUUID binary:  00 84 0e 55  9b e2  d4 41  a7 16 44 66 55 44 00 00
                [reversed]   [rev]  [rev]  [preserved as-is]

UUID binary:    55 0e 84 00  e2 9b  41 d4  a7 16 44 66 55 44 00 00
                [as-is throughout]

Querying with CSUUID¶

When using MCP MongoDB tools, mongosh, or Compass:

// ❌ WRONG - Standard UUID won't match existing documents
db.pmStudy.find({ _id: UUID("550e8400-e29b-41d4-a716-446655440000") })

// ✅ CORRECT - Use CSUUID function
db.pmStudy.find({ _id: CSUUID("550e8400-e29b-41d4-a716-446655440000") })

// ✅ CORRECT - Direct BinData with subtype 3
db.pmStudy.find({ _id: BinData(3, "AIQOVZvi1EGnFkRmVUQAAA==") })

Working with GUIDs in Code¶

Always let the MongoDB C# driver handle serialization:

// ✅ CORRECT - Let the driver serialize
var studyId = Guid.Parse("550e8400-e29b-41d4-a716-446655440000");
var study = await collection.Find(s => s.Id == studyId).FirstOrDefaultAsync();

// ❌ WRONG - Manual binary conversion will likely use wrong byte order
var bytes = studyId.ToByteArray();
var bsonBinary = new BsonBinaryData(bytes, BsonBinarySubType.UuidLegacy);

Driver Connection Behaviour¶

`directConnection=true` for single-server local / TestContainers setups¶

What the code does

When ConnectionStrings:MongoConnection:ClusterAddresses resolves to a single server, both the connection-string builder in MongoConnectionSettings.ConnectionString and the MongoClientSettings builder in MongoContext.BuildSettingsFromClusterAddresses set directConnection=true. Multi-server (real replica set) and SRV (ClusterAddress) paths are untouched.

Why

The local-dev docker-compose.dev.yml runs mongo:7 with --replSet rs0 and initialises the replica set using members:[{_id:0, host:'localhost:27017'}]. That localhost:27017 is the mongo process's internal address inside the container. When the container is published on a different host port (as in side-by-side worktrees, where each worktree gets its own port — e.g. PORT_MONGODB=29585), connecting from the host to localhost:29585 succeeds, but then the MongoDB driver performs SDAM topology discovery: it asks the server for replica-set membership, the server reports localhost:27017, and the driver replaces the configured endpoint with that self-advertised address. The host has nothing on 27017, so every subsequent operation times out with "Connection refused" against Unspecified/localhost:27017.

directConnection=true pins the driver to the configured host:port, skipping SDAM replacement. Change streams still work because the server is genuinely part of a replica set — the flag only affects client-side topology discovery, not server capabilities.

Symptoms if this regresses

Service startup logs timeouts of this shape, even though appsettings.local.json clearly points at the right port:

TimeoutException: ... selecting a server ... Client view of cluster state is
{ ClusterId : "N", Type : "ReplicaSet", State : "Disconnected",
  Servers : [{ EndPoint: "Unspecified/localhost:27017", ... }] }

The give-away is Type : "ReplicaSet" combined with localhost:27017 when you configured a different port — that's the self-advertised address winning over your settings.

Alternative fixes considered

Changing mongo's internal port to match the host-mapped port (as e2e/docker-compose.yml does, with --port 27018 + 27018:27018) also works but requires docker volume reset when the port changes and complicates side-by-side worktrees where each picks a different port. The directConnection flag avoids all of that.

Scope

Tests: MongoContextTests.BuildMongoClientSettings_WithClusterAddresses_* and MongoConnectionSettingsTests.ConnectionString_* pin the behaviour: single server → DirectConnection=true / ?directConnection=true; multi-server → neither.

MongoContext.cs - Collection naming implementation
MongoUtils.cs - GUID serialization configuration
MongoConnectionSettings.cs - Connection string construction