Contain and verify an incident
Incident response contains active exposure or failure, restores the affected capability and records what was actually verified. The responsible technical owner coordinates recovery and any customer or privacy communication. A monitoring alert starts investigation; its absence or closure does not prove the business outcome recovered.
Prerequisites and authority
Section titled “Prerequisites and authority”Identify the affected environment, component, actor/resource scope and observed impact. Use the current incident runbook and the relevant release, credential or recovery owner. Maintain a safe incident record with times, sanitized symptoms and evidence locations. Credentials, private payloads and customer personal data do not belong in that record.
Use the independent alert/contact route for data exposure, lost company isolation or complete outage. If Control Center or the normal alert route is unavailable, use the available approved record and owner contact path; restoring those tools is part of recovery. This documentation does not grant permission to send messages or trigger real alert deliveries.
Response procedure
Section titled “Response procedure”- Contain the active exposure or failure through the smallest authorized measure. Preserve useful diagnostic evidence before changing state where possible. Avoid an indiscriminate restart, data cleanup or credential revocation that destroys the only recovery path.
- Record start time, affected systems and observed impact. Distinguish confirmed exposure or failure from a suspected risk. Note current API/frontend versions and relevant schema/provider state without copying sensitive records.
- Notify the responsible technical owner through the appropriate route. Independent alerting is necessary when the application reporting path may itself be broken; a database outage can also prevent Control Center incident creation.
- Diagnose the confirmed failure boundary using workflow diagnosis. Select the owning correction: compatible application rollback, verified credential replacement/rotation, forward schema correction or explicitly authorized isolated recovery.
- Verify the affected behaviour and the negative boundary at the corrected version. Check component health/readiness as prerequisites, then the named actor/resource outcome. Reconcile uncertain provider effects before retrying them.
- Record the recovery observation, remaining limitations and follow-up owner. The responsible person decides required customer/privacy communication. Preserve original receipts rather than rewriting them to match the successful result.
Containment and recovery reference
Section titled “Containment and recovery reference”| Incident class | Owning response boundary | Verification needed |
|---|---|---|
| Application publication failure | Guarded deployment and compatible previous release | Actual current link/version, readiness and changed behaviour |
| Required credential invalid | Broker/configuration and replacement procedure | Replacement usable before revocation, runtime consumes it, affected function works |
| Lost company isolation | Stop exposure and preserve restricted evidence | Negative cross-company test plus intended authorized access |
| Partial schema publication | Actual ledger/catalogue and forward-only handling | Compatible writer and affected scenario, not blind restart |
| Suspected lost data/objects | Backup generation and authorized recovery owner | Component integrity plus identity/application scenario |
| Provider timeout/unknown result | Durable operation and destination identity | Reconciled external outcome before replay |
Reporting and sensitive diagnostics
Section titled “Reporting and sensitive diagnostics”The incident reporter uses allowlisted event codes and bounded metadata. That does not mean all runtime logs are automatically redacted. Raw exceptions, provider bodies or free-text details may require restricted handling. Logs and evidence owns those limits; do not copy raw diagnostics into a public issue or review artifact.
Watchdog state distinguishes observation from confirmed alert delivery. A healthy probe after an outage is a component recovery signal, not proof that every queued action resumed or that a real recipient received a message. Retain that uncertainty until the relevant record outcome is checked.
Synthetic example and completion
Section titled “Synthetic example and completion”After a synthetic cross-company access regression, contain the affected route, preserve the denied/allowed test setup and publish the scoped correction. Verify that Company A cannot read Company B’s record and that the intended Company A actor still can. A passing process-health check supplies no isolation evidence.
Recovery, publication and external recovery define their own rollback limits. Close the incident only with the actual affected result and a concrete owner for any residual gap; documentation completeness or a green test unrelated to the failure cannot substitute for that result.