Skip to content

Health, readiness and runtime progress

Runtime signals establish bounded facts about availability, dependencies, identity or progress. They are useful together, but none establishes that an invoice, notification or payroll handoff reached its final business outcome. Stalled work follows those record-level outcomes.

Signal Actual contract Limit
/healthz API returns status: ok No database, Storage or worker proof
/readyz 200 ready or 503 not_ready with dependency/component checks Not authenticated actor/resource or provider acceptance
API /version Source SHA from the deployed release stamp, or unknown Static product-version label is not the deployed commit
Frontend version.json Published browser build identity Does not identify API or applied schema
Heartbeat Expected component health, timestamp and progress-dependent predicates Freshness does not prove a particular queued record completed
Local watchdog Local readiness plus configured systemd-service checks Cannot independently detect every total host/network outage
External uptime workflow Public frontend/API/version/readiness probes from outside AX42 No signed-in business journey

Readiness first checks required configuration and that the Supabase REST gateway returns a successful status. An unauthorized, forbidden or missing gateway path is a failure, not evidence of reachability sufficient for readiness. It then queries the PII-free runtime_heartbeats sentinel through a dedicated bounded client: gateway availability alone does not prove PostgreSQL can execute queries.

Component health follows when the database is usable. The Storage check lists at most one entry from project-files. If the database dependency fails, components and Storage are explicitly marked unavailable instead of omitted or repeatedly probed through the same dead path. Public remediation uses curated messages rather than raw provider exceptions.

The component heartbeat interval is 30 seconds and maximum heartbeat age is 90 seconds. Expected components derive from configuration and enabled schedulers. Overrides must include API, reject unknown/empty components and preserve the paired Fortnox scheduler/workers contract. Completed-progress trackers also matter: an APScheduler object marked running can remain alive while every tick fails.

The deliberate API-only runtime disables schedulers. Separately, TOOLORA_DISABLE_RUNTIME_HEARTBEATS permits the documented canary/test omission of shared heartbeat writes; readiness labels that component check skipped while still checking actual dependencies. Do not interpret that result as evidence of normal background processing, or copy the bypass into the normal runtime to conceal a failure.

Readiness incident reporting is bounded and uses a fixed incident code. When the database dependency is unavailable it avoids another reporting request through the dead path. Absence of a Control Center incident is therefore not evidence of health. Log and incident evidence describes safe diagnostics.

The local watchdog timer starts after two minutes and runs every five minutes. Its probe has a 12-second whole-operation deadline; SMTP has a 25-second deadline and 15-second socket timeout. It uses separate alert credentials and addressing configuration, preserves retryable state and alerts after two consecutive failed probes. A failed alert delivery is not recorded as confirmed delivery.

The external workflow runs every five minutes on a GitHub-hosted runner and serializes its incident workflow. That serialization does not cover arbitrary manual script runs. Neither schedule proves an exact detection time under runner or network delay.

For a synthetic delayed notification, record /version, readiness component results and the work row’s last confirmed state. A healthy API with a stale notification tracker points to that component; a fresh tracker with an individual failed row needs record/provider diagnosis. Diagnosis defines the next step. Runtime services and test levels keep declared topology, mocked monitor tests and current public observations distinct.