Skip to content

Test levels and selected checks

Toolora combines deterministic unit/static checks, local database and token-policy tests, component browser checks and scoped functional verification. These levels answer different questions. A test’s existence, a coverage label and a successful execution on a particular commit are separate facts.

make ci-plan BASE=origin/main selects shared local commands from the complete diff. Before marking a PR ready, inspect that plan and run the relevant make ci-* targets. make check is a useful local subset; it is not the complete set of CI surfaces.

Surface Principal checks Evidence limit
Security Current/history PII and sanitation checks, pinned Gitleaks, policy checks Finds covered patterns; not a universal absence-of-secrets or retention certification
Backend Static contract checks and hermetic pytest Synthetic inputs/mocks do not prove a provider completed an effect
Frontend Type/lint/Vitest, Storybook browser, email rendering and builds Stories do not establish every authenticated journey or visual/accessibility outcome
Database Migration guards, clean build, SQL/RLS, seed, real-token integration, MCP DB, types and upgrade Local/disposable database evidence, not shared business acceptance
Infrastructure and shell Ansible/policy/broker contracts, ShellCheck and Bats release/schema guards Temporary fixtures/fake dependencies do not operate AX42
Windows Separate PowerShell contracts on the required platform A Linux-only run cannot substitute for it
Dependency audit Runtime dependency vulnerability checks Snapshot at the checked lockfiles and advisory state
Handbook Astro/type/build, whole-site verification, Chromium and packaged HTML Documentation behaviour and distribution; no product-flow certification

Draft PRs defer application suites while retaining security and the aggregate result. Ready PRs run the selected surfaces. Unknown paths and changes to central workflow/Make/selection logic broaden selection so shared policy cannot quietly avoid its own checks. Relevant database work includes both clean-schema and upgrade rehearsal against the declared published prefix.

The required CI result checks selected job outcomes. Missing, cancelled, failed or unexpectedly skipped selected jobs are not green delivery. Do not deselect a check, lower a baseline, add a skip marker or remove protection to reduce consumption. An unchanged transient failure permits retrying its failed jobs after reading the failed step; deterministic failures require a correction.

A PR run can test GitHub’s combined merge commit rather than the branch head used to build a documentation archive. Record the actual checkout and compare relevant inputs before reusing results. An old successful run does not become a fresh result simply because the branch still exists.

Local database tests use repository-owned disposable targets. Backend tests must not discover shared runtime dotenv or provider configuration. Component fixtures explicitly fake HTTP/provider effects where their contract says so. Reading an assertion is source review; executing it establishes a result only within that fixture.

The test registry records scenarios and coverage depth. Labels such as specified, partial, automated, operator or retired describe coverage, not a current pass. CI does not execute every attended persona journey. A browser test against Vi Målar needs the right actor and external-effect boundary; a successful invoice preview cannot stand in for real accounting acceptance.

A synthetic release test switches temporary frontend directories and makes fake HTTP fail. It can prove restoration of the previous compatible link. It does not prove the real proxy, service identity or provider path. Pair it with the relevant publication observation when application code changes.

When a check fails, preserve its step, checked commit and target before changing inputs. Update a ready PR back to draft while fixing it, then rerun the affected checks. Release procedure, health and evidence explain how local results, runtime observations and business outcomes fit into a truthful delivery statement.