Health, readiness and runtime progress
Runtime signals establish bounded facts about availability, dependencies, identity or progress. They are useful together, but none establishes that an invoice, notification or payroll handoff reached its final business outcome. Stalled work follows those record-level outcomes.
Signal reference
Section titled “Signal reference”| Signal | Actual contract | Limit |
|---|---|---|
/healthz |
API returns status: ok |
No database, Storage or worker proof |
/readyz |
200 ready or 503 not_ready with dependency/component checks |
Not authenticated actor/resource or provider acceptance |
API /version |
Source SHA from the deployed release stamp, or unknown | Static product-version label is not the deployed commit |
Frontend version.json |
Published browser build identity | Does not identify API or applied schema |
| Heartbeat | Expected component health, timestamp and progress-dependent predicates | Freshness does not prove a particular queued record completed |
| Local watchdog | Local readiness plus configured systemd-service checks | Cannot independently detect every total host/network outage |
| External uptime workflow | Public frontend/API/version/readiness probes from outside AX42 | No signed-in business journey |
Readiness order and failure
Section titled “Readiness order and failure”Readiness first checks required configuration and that the Supabase REST gateway returns a successful status. An unauthorized, forbidden or missing gateway path is a failure, not evidence of reachability sufficient for readiness. It then queries the PII-free runtime_heartbeats sentinel through a dedicated bounded client: gateway availability alone does not prove PostgreSQL can execute queries.
Component health follows when the database is usable. The Storage check lists at most one entry from project-files. If the database dependency fails, components and Storage are explicitly marked unavailable instead of omitted or repeatedly probed through the same dead path. Public remediation uses curated messages rather than raw provider exceptions.
The component heartbeat interval is 30 seconds and maximum heartbeat age is 90 seconds. Expected components derive from configuration and enabled schedulers. Overrides must include API, reject unknown/empty components and preserve the paired Fortnox scheduler/workers contract. Completed-progress trackers also matter: an APScheduler object marked running can remain alive while every tick fails.
Canary scope and reporting
Section titled “Canary scope and reporting”The deliberate API-only runtime disables schedulers. Separately, TOOLORA_DISABLE_RUNTIME_HEARTBEATS permits the documented canary/test omission of shared heartbeat writes; readiness labels that component check skipped while still checking actual dependencies. Do not interpret that result as evidence of normal background processing, or copy the bypass into the normal runtime to conceal a failure.
Readiness incident reporting is bounded and uses a fixed incident code. When the database dependency is unavailable it avoids another reporting request through the dead path. Absence of a Control Center incident is therefore not evidence of health. Log and incident evidence describes safe diagnostics.
Independent monitors
Section titled “Independent monitors”The local watchdog timer starts after two minutes and runs every five minutes. Its probe has a 12-second whole-operation deadline; SMTP has a 25-second deadline and 15-second socket timeout. It uses separate alert credentials and addressing configuration, preserves retryable state and alerts after two consecutive failed probes. A failed alert delivery is not recorded as confirmed delivery.
The external workflow runs every five minutes on a GitHub-hosted runner and serializes its incident workflow. That serialization does not cover arbitrary manual script runs. Neither schedule proves an exact detection time under runner or network delay.
Diagnostic example and verification
Section titled “Diagnostic example and verification”For a synthetic delayed notification, record /version, readiness component results and the work row’s last confirmed state. A healthy API with a stale notification tracker points to that component; a fresh tracker with an individual failed row needs record/provider diagnosis. Diagnosis defines the next step. Runtime services and test levels keep declared topology, mocked monitor tests and current public observations distinct.