Provisioning

The Provisioning module keeps downstream systems continuously synchronized with the authoritative Study Model. It is organized around Crosswalk Links — durable entity-level links between Study entities and remote entities (records in collections on remote systems: BSI, Infor, Smartsheet) — and a desired-state reconciliation loop that provisions approved model state outward as soon as it changes, verifies that each write actually propagated, and flags any later remote-side change as drift. LIMS consumes approved study data through a read-only API and is not synchronized.

Mockup

Study detail – Provisioning tab

/studies/:id/provisioning – Placeholder tab within the Study detail view. Not yet fully represented in the mockup.

The provisioning model

Crosswalk Links

Remote systems will contain data (BSI study shells, Infor projects, Smartsheet rows, …) that should be kept in sync with the Study Model. Each Study Entity will be mapped to remote entities via a "CrosswalkLink".

The module provides implementations for multiple Crosswalks which define how a remote entity is mapped from a Study Entity into records on a remote system, and vice versa, as well as policies with regards to eagerness to overwrite and/or automatically re-provision upon encountering drift.

A CrosswalkLink is the persisted, instance-level link: one Study entity ↔ one remote entity (system / collection / remote id), keyed to its crosswalk in the registry, with provenance recording how the link was established — matched by the identity rule, created by Study Setup, or manually linked.

Each link stores a reference to a Study Entity, the specific Checkpoint it has provisioned up to, and a crosswalk key.

Additionally, each CrosswalkLink stores three entity states: Desired, Applied, and Observed.

Value	Meaning	Updated when
Desired	What the remote entity should look like	A new Checkpoint is committed, the state is updated from the Study Model
Applied	The last entity state we confirmed was provisioned to the remote	The Provisioner confirms a successful update on the remote system, having published the desired state.
Observed	The remote entity as we last saw it	When the Provisioner confirms a successful update on the remote system, it sets Observed = Applied. Whenever the remote is polled or manually refreshed, the Provisioner captures and updates the Observed state.

The CrosswalkLink's provisioning_status is derived from comparing the three values:

Status	Condition	Note
Provisioned	Desired = Applied = Observed	The remote matches what we last applied, which matches the model
Drifted	Desired = Applied ≠ Observed	The remote moved while we had nothing pending
Converged	Desired = Observed ≠ Applied	The remote matches what we desire, even if the Provisioner did not perform the update
Pending	Desired ≠ Applied = Observed	A new desired state awaits provisioning
Conflicted	All three differ	The remote moved while a change was pending

stateDiagram-v2
    [*] --> Pending : link established
    Pending --> Pending : attempt failed<br/>(retry / escalate)
    Pending --> Provisioned : guarded write confirmed<br/>by read-back
    Pending --> Converged : remote independently<br/>reached desired state
    Pending --> Conflicted : remote moved while pending<br/>(invalidated write or observation)
    Converged --> Provisioned : read-back confirms<br/>desired already landed
    Provisioned --> Pending : checkpoint changes desired
    Provisioned --> Drifted : remote moved
    Drifted --> Provisioned : update_remote (re-provision)
    Drifted --> Pending : update_model (via changeset)
    Drifted --> Conflicted : checkpoint changes desired<br/>while drifted
    Conflicted --> Pending : resolved per conflict policy

The Provisioner

A single eager loop turns Pending links into Provisioned ones. There is no persisted work queue — the set of Pending links is the work list:

• Eager scheduling. A policy on StudyModelUpdated wakes the Provisioner immediately; a periodic sweep is the safety net for anything missed. It always pulls the next provisionable change from the Pending links, ordered by how long the change has been waiting and/or other prioritization policies (to be defined), respecting per-link retry backoff.
• Volatile execution. Each provisioning attempt is a Celery task carrying no durable state of its own. At execution time it reads the link's current Desired and Applied and performs a guarded write — the write proceeds only if the remote stayed still relative to Applied (the remote's native conditional-update mechanism where one exists, read-compare-write otherwise) — followed by a read-back that refreshes Observed. The operation (create, update, or decommission of the remote entity) follows from the link and its crosswalk.
• Materialized results. Outcomes are recorded as provisioning events in the Audit Trail and on the link itself: every attempt updates latest_attempt (timestamp, outcome, error context), and a verified success advances Applied and applied_on.

An attempt ends in exactly one outcome:

Outcome	Meaning	Effect on the link
Confirmed	The write was accepted and the read-back shows the desired state landed	Applied and applied_on advance → Provisioned
Failed (retryable)	Transient error — timeout, connectivity, throttling	latest_attempt recorded; stays Pending, retried with exponential backoff
Failed (conclusive)	The remote rejected the write definitively (validation, permissions)	latest_attempt recorded; escalated for human resolution — retry, reassign, or authorized manual completion
Invalidated	The precondition did not hold: the remote moved	The fresh Observed state is recorded on the link, which reclassifies it as Conflicted — or Converged if the remote already matches Desired

CrosswalkLink lifecycle

1. A new desired state. Values enter the Study Model only through the governed change flow: an approved changeset produces a new Checkpoint, which is projected through each crosswalk's field map, updating Desired on every affected link. Only Active studies participate — Draft and Archived studies are silent.
2. Automatic provisioning, ASAP. The Provisioner picks up the now-Pending link and executes a guarded write as a volatile attempt.
3. Verify. The remote's acknowledgment is never trusted on its own: only when the read-back shows the desired state landed do Applied and applied_on advance and the link become Provisioned. Until then the link remains Pending (write accepted, read-back outstanding) — never a false "provisioned".
4. Afterwards, monitoring. Scheduled polls and manual refreshes keep updating Observed. Either the remote stays still — the link remains Provisioned — or it moves: Drifted, raising DriftDetected in the study's Attributes view, resolved explicitly (update_remote / update_model / mute, see Studies) and never overwritten automatically.

Resilience

Connectivity to remote systems is assumed to be unreliable. The model handles failures as follows:

• Guarded writes — every attempt checks its precondition at write time. A remote change not yet seen by polling invalidates the attempt rather than being overwritten.
• Verified writes — a write counts as provisioned only after read-back confirmation, not on API acknowledgment. A write that times out but lands is recognized on the invalidation read-back and resolves to Provisioned.
• Stateless work list — the work list is the set of Pending links; each attempt reads the link's current values at execution time. Crashes, lost tasks, and duplicate deliveries converge to the same state, and successive model changes coalesce into the net difference.
• Freshness windows — Provisioned is reported as Unverified when no successful observation has occurred within the system's freshness window.
• Retry, then escalate — retryable failures are retried with exponential backoff (tracked on latest_attempt); exhausted retries and conclusive rejections are escalated for human resolution (retry, reassign, or authorized manual completion).
• Status rollups — link statuses aggregate per remote system and per study (e.g. "2 links pending to BSI, 1 drifted on Smartsheet").

Write unit and atomicity

Each provisioning attempt targets one remote entity (one link), applying the full desired state in a single guarded write. Whether several links touched by the same checkpoint must land atomically on a remote system ("no partial pushes") depends on transactional capabilities of each remote system.

The crosswalk therefore models its entity at the remote's atomic write granularity: a set of records where the remote can update multiple records in a single request, a single record otherwise.

Entities

The data model this module introduces and persists:

Entity

Description

CrosswalkLink

Entity-level link between one Study entity and one remote entity (system / collection / remote id), carrying its crosswalk key (referencing the code-defined crosswalk registry — never the field map itself) and link provenance (matched / created / manual). Stores the three entity states — Desired, Applied (with applied_on), Observed (with its capture timestamp) — plus latest_attempt (timestamp, outcome, error context of the most recent provisioning attempt). Applied is the one record that cannot be re-derived; Desired and Observed are refreshed from the current Checkpoint and the latest observation. provisioning_status is always derived from the three values.

Use Cases

• Link a Study entity to an existing remote entity through a crosswalk's identity rule, create the remote record where the crosswalk permits, or link manually.
• Automatically provision approved Study Model changes to remote systems as soon as a checkpoint is committed, prioritizing the longest-waiting pending changes.
• Execute every provisioning attempt as a guarded write whose precondition detects remote movement at write time; reclassify invalidated writes from the fresh Observed state instead of overwriting the remote change.
• Confirm by read-back that every provisioned state actually landed; never report a link as provisioned from an acknowledgment alone.
• Recognize convergence — the remote already matches the desired state — and adopt it as Provisioned without a redundant write.
• View link status rolled up per remote system and study — including the checkpoint each link has provisioned up to, last verified at timestamps, and attribute-level diffs on drifted/conflicted links.
• Refresh each link's Observed state on post-write read-back, scheduled poll, and manual refresh.
• Detect drift (the remote moved while nothing was pending) and surface it for explicit resolution: update_remote, update_model, or mute.
• Detect conflicts (the remote moved while a change was pending) and halt provisioning until resolved per the conflict policy.
• Retry and manually complete failed provisioning attempts with appropriate authorization.
• Expose approved study data to LIMS via API (read-only query interface).

Notes

• Only approved checkpoints of Active studies are synchronized; Draft and Archived studies trigger no provisioning actions.
• Checkpoint approval is the only release gate. The previous coordinated "Release Package" and scheduled-release-window concepts are removed in favor of continuous reconciliation from the moment a study is Active.
• Drift is never overwritten automatically; resolution flows through the Studies drift workflow (DriftDetected / DriftResolved / DriftMuted).
• A link can become Provisioned without Study Setup ever writing: a Converged link (remote already matches the desired state) is adopted — Applied advances to the observed state — and is never treated as drift.
• Smartsheet write-back is the same mechanism as BSI/Infor provisioning: push-only links over the allowed-field crosswalk. Smartsheet is not authoritative for the study model; the former SmartsheetMismatchDetected is subsumed by DriftDetected.
• The former system-specific release events (BSIProvisioned, InforProvisioned, SmartsheetWriteBackPerformed) are subsumed by the generic provisioning events below, which carry the target system as an attribute; the field scopes that were previously enforced at the requirement level are now defined per-system via its crosswalks.
• LIMS access is read-only; LIMS queries specific Study Setup fields via a governed API, not a full data dump.
• Provisioning attempts are volatile: outcomes are materialized in the Audit Trail and on the link (latest_attempt, applied_on), while provisioning_status is always re-derived from the Desired/Applied/Observed comparison — lost, duplicated, or stale tasks cannot corrupt status.
• A Provisioned claim is only as fresh as its last observation; beyond the freshness window it degrades to Unverified.

Events

Events from the remote_sync_provisioning bounded context:

Event	Description
RemoteEntityLinked	A CrosswalkLink was established between a Study entity and a remote entity (matched, created, or manually linked)
RemoteEntityCreated	Study Setup created a new record in a remote collection, per the crosswalk's creation policy
ProvisioningApplied	A guarded write was accepted by the remote system (not yet verified)
ProvisioningConfirmed	A read-back verified the desired state landed (or a Converged link was adopted); Applied / applied_on advanced and the link is Provisioned
ProvisioningFailed	An attempt failed — transiently (will retry), conclusively, or by read-back mismatch — recorded on latest_attempt
ProvisioningInvalidated	The precondition no longer held at write time; the fresh Observed state reclassified the link (Conflicted, or Converged if the remote already matches the desired state)
ProvisioningEscalated	A link exhausted retries or failed conclusively and was escalated for human resolution
ProvisioningFailureResolved	An escalated failure was resolved (successful retry or authorized manual completion)
RemoteStateObserved	A link's Observed state was refreshed (post-write read-back, scheduled poll, or manual refresh)
ConflictDetected	The remote moved while a change was pending; provisioning is halted pending resolution
LIMSDataQueried	LIMS queried permitted Study Setup data via API

Drift itself (DriftDetected, DriftResolved, DriftMuted) is owned by the Studies module; the Provisioner's observations are its trigger.

Pending definitions

• [ ] Conflict policy — what happens when a link becomes Conflicted: halt for human review vs. overwrite (model wins). Either way the remote's intervening state is preserved in the Audit Trail (via the observation that detected it). Declared per crosswalk in the registry (so changing it is a release); whether Drifted gets a similar knob (flag-only vs. auto-re-provision) is part of the same business decision.
• [ ] Provisioner prioritization — the ordering of the pending work list (wait time, study/system priority) and the retry/backoff parameters.
• [ ] Remote write atomicity — each remote system's multi-record transactional capabilities, which set the granularity of crosswalk entities (see Write unit and atomicity): where multi-record updates are atomic, one crosswalk entity may span a set of records.
• [ ] Entity existence handling — what triggers link establishment when a checkpoint introduces a new crosswalked entity; the per-crosswalk decommission policy for the decommission operation (deactivate remotely, flag-only, or forbid — auto-deletion of remote records is assumed unsafe); detection of orphaned links (remote record deleted) and unmanaged in-scope remote records (created outside Study Setup).
• [ ] Per-system guarded-write support — which remote systems offer native conditional updates (ETag/version) and where read-compare-write is the fallback.
• [ ] Per-system cadence — polling schedules and freshness windows for each remote system.
• [ ] Participating entities and concrete crosswalks — which Study entities bind to which remote collections, and their field maps (next iteration; see Crosswalks).

Remote Systems

Study Setup communicates programmatically with other systems within PMG. This document lists the remote systems involved in the implementation along with implementation notes and connection information required for each of the Development, UAT and Production deployments.

Infor

Infor is a Finance/ERP platform used within PMG.

Study Setup will provision project structure information including kit configurations in Infor. Inbound study data reaches Study Setup through a replication feed delivered to an AWS S3 bucket; write operations back to Infor go through its API (details being worked out with Kyle Tucker in a separate thread).

Open points to confirm with PMG:

• Whether to stand up separate dev / uat / prod Infor environments, use a single shared Infor instance, or take a different approach altogether.
• A dev / uat environment to exercise write operations against, with the replication feed also set up for it — so far only the prod replication feed has been discussed.
• For each environment: the replication bucket name / ARN and specs or examples of the replication data format.

Environment	Replication Bucket (name / ARN)	API Endpoint
dev	TBD	TBD
uat	TBD	TBD
prod	TBD (replication feed agreed, details pending)	TBD

BSI

Lab/Specimen system.

Study Setup will provision study/specimen shell information into BSI. Dev/UAT and prod are distinct endpoints at different addresses but share the same credentials (username Sebastian.Acuna).

Environment	Address	Username	Password
dev	https://pfm-mirror-rest.bsisystems.com/api/rest/PRECISION	Sebastian.Acuna	***********************************y2zh
uat	https://pfm-mirror-rest.bsisystems.com/api/rest/PRECISION	Sebastian.Acuna	***********************************y2zh
prod	https://pfm-rest.bsisystems.com/api/rest/PRECISION	Sebastian.Acuna	***********************************y2zh

Smartsheet

Project tracking and coordination worksheet.

Study Setup will publish study information into the project worksheet.

For dev and uat, the "1 - Project Tracker" sheet has been cloned into DEV and UAT copies in a workspace we created ourselves. Dev and uat read from and write to those clones, which are re-cloned from production as necessary. API tokens for the cloned sheets are self-provisioned through the Smartsheet UI.

Only prod writes to the production sheet, using an API token to be provided further ahead.

Environment	Address	Sheet ID	API Key
dev	https://api.smartsheet.com/2.0/	M8QhmPg3m2cxMpcRW7r66w7gVQHFGQWhvvcq26w1 (DEV clone)	*********************************NJtX (self-provisioned)
uat	https://api.smartsheet.com/2.0/	mFf2Fh79JR6vxqGw326XqQGwvpj9fFFXrHH9f2f1 (UAT clone)	*********************************NJtX (self-provisioned)
prod	https://api.smartsheet.com/2.0/	(production sheet)	TBD

ZenQMS

Document control/quality management system.

Study Setup will only read from ZenQMS in order to discover SOP (Standard Operating Porcedure) documents.

The production ZenQMS environment will be used for all environments, with the same readonly credentials.

Environment	Address	API Key
dev, uat & prod	https://public-api.zenqms.com/v2	(secret)

DocuSign

E-signature platform.

We suggest creating a dedicated Application for development / testing, with an API Key associated to that dev/test application. Some testing can be run against the DocuSign mock server, but final validation will require the dedicated app.

Environment	Address	API Key
dev, uat & prod	https://api.signnow.com	TBD (dev/test application)