Skip to content

Stalled work and diagnosis

A stalled workflow is a named item that has not reached its expected next state. An idle worker can be healthy when no eligible work exists. A live scheduler can be unhealthy when every tick fails. Diagnose component liveness, completed worker progress and business-item state separately before restarting or retrying anything.

Signal What it establishes What remains to check
API responds/health endpoint passes The process can serve that check Database, required components and the affected workflow
APScheduler reports running Its scheduler is alive in this process Whether required jobs executed successfully
WorkerProgress is fresh Required registered workers completed ticks within their bounds Whether a particular source, command or item progressed
Current-instance component heartbeat is fresh That runtime instance reported the expected component state Exact business receipt and provider/source coverage
Pending/deferred item A workflow still has work or a precondition Eligibility, lease, retry time, source verification and budget
Provider receipt/readback The named external observation exists Whether it matches the approved identity/value and the workflow’s completion definition

WorkerProgress stores code-defined job names and timestamps, not tenant records, exception messages or provider payloads. A required worker is fresh within three of its configured cadences, with a 30-second floor; future timestamps are not accepted as freshness. A new process/start gets a fresh tracker rather than inheriting another run’s successful tick. Missing, skipped or failed ticks must not manufacture progress.

  1. Identify the company, resource and owning workflow. Read the item’s current status and last confirmed event/receipt; avoid private payloads in artifacts.
  2. Check global scheduler and component enablement, current instance and expected component configuration. An intentionally disabled service is different from a registered worker that never fires.
  3. Check completed tick progress, current-instance heartbeat and the owning error category. A stale component needs diagnosis before a restart becomes evidence.
  4. Inspect the item’s claim/lease, retry eligibility/time and any concurrent processing. Process-local locks do not replace a durable lease.
  5. Check source attestation, original verification, model/feature budget and provider command state. A safe deferral can coexist with healthy tick progress.
  6. Use the workflow’s supported recovery and verify the named result. Preserve any missing coverage or unresolved decision rather than clearing it cosmetically.

The scheduler inventory identifies registration and cadence. Intent and outcomes identifies durable work. Runtime health and incident procedures retain their own operational owners; this reference does not replace their release/recovery commands.

A synthetic mail tick can complete successfully while Candidate A waits for a verified copy or tomorrow’s interpretation budget. Inspect its safe error/status and current source coverage. Repeatedly queuing the same AI work does not create missing bytes or a larger budget; human manual classification remains a separate admitted path when its preconditions hold.

Synthetic Export B can remain unknown after a provider timeout. A green heartbeat or worker restart does not prove its financial effect failed. Reconcile the existing command before another submission. Conversely, a scheduler running while its claim query throws on every tick must not report successful worker progress.

After correction, verify a new completed tick and the affected item’s intended transition/receipt. A successful restart alone proves only that restart. Source coverage, provider value and current reader authorization need their own checks. Never mark leases, exports or review candidates completed by hand to improve health output, and never test a real accounting write in Vi Målar.

Progress/heartbeat tests use synthetic clocks, isolated app state and substituted workers/database responses; actual SQL claims and provider receipts have different scope. The delivery PR records actual checks and reused evidence. No shared business item was retried or changed for this documentation review.