Test levels and selected checks
Toolora combines deterministic unit/static checks, local database and token-policy tests, component browser checks and scoped functional verification. These levels answer different questions. A test’s existence, a coverage label and a successful execution on a particular commit are separate facts.
make ci-plan BASE=origin/main selects shared local commands from the complete diff. Before marking a PR ready, inspect that plan and run the relevant make ci-* targets. make check is a useful local subset; it is not the complete set of CI surfaces.
Check reference
Section titled “Check reference”| Surface | Principal checks | Evidence limit |
|---|---|---|
| Security | Current/history PII and sanitation checks, pinned Gitleaks, policy checks | Finds covered patterns; not a universal absence-of-secrets or retention certification |
| Backend | Static contract checks and hermetic pytest | Synthetic inputs/mocks do not prove a provider completed an effect |
| Frontend | Type/lint/Vitest, Storybook browser, email rendering and builds | Stories do not establish every authenticated journey or visual/accessibility outcome |
| Database | Migration guards, clean build, SQL/RLS, seed, real-token integration, MCP DB, types and upgrade | Local/disposable database evidence, not shared business acceptance |
| Infrastructure and shell | Ansible/policy/broker contracts, ShellCheck and Bats release/schema guards | Temporary fixtures/fake dependencies do not operate AX42 |
| Windows | Separate PowerShell contracts on the required platform | A Linux-only run cannot substitute for it |
| Dependency audit | Runtime dependency vulnerability checks | Snapshot at the checked lockfiles and advisory state |
| Handbook | Astro/type/build, whole-site verification, Chromium and packaged HTML | Documentation behaviour and distribution; no product-flow certification |
Automatic selection and mandatory result
Section titled “Automatic selection and mandatory result”Draft PRs defer application suites while retaining security and the aggregate result. Ready PRs run the selected surfaces. Unknown paths and changes to central workflow/Make/selection logic broaden selection so shared policy cannot quietly avoid its own checks. Relevant database work includes both clean-schema and upgrade rehearsal against the declared published prefix.
The required CI result checks selected job outcomes. Missing, cancelled, failed or unexpectedly skipped selected jobs are not green delivery. Do not deselect a check, lower a baseline, add a skip marker or remove protection to reduce consumption. An unchanged transient failure permits retrying its failed jobs after reading the failed step; deterministic failures require a correction.
A PR run can test GitHub’s combined merge commit rather than the branch head used to build a documentation archive. Record the actual checkout and compare relevant inputs before reusing results. An old successful run does not become a fresh result simply because the branch still exists.
Local versus attended evidence
Section titled “Local versus attended evidence”Local database tests use repository-owned disposable targets. Backend tests must not discover shared runtime dotenv or provider configuration. Component fixtures explicitly fake HTTP/provider effects where their contract says so. Reading an assertion is source review; executing it establishes a result only within that fixture.
The test registry records scenarios and coverage depth. Labels such as specified, partial, automated, operator or retired describe coverage, not a current pass. CI does not execute every attended persona journey. A browser test against Vi Målar needs the right actor and external-effect boundary; a successful invoice preview cannot stand in for real accounting acceptance.
Synthetic example and failure handling
Section titled “Synthetic example and failure handling”A synthetic release test switches temporary frontend directories and makes fake HTTP fail. It can prove restoration of the previous compatible link. It does not prove the real proxy, service identity or provider path. Pair it with the relevant publication observation when application code changes.
When a check fails, preserve its step, checked commit and target before changing inputs. Update a ready PR back to draft while fixing it, then rerun the affected checks. Release procedure, health and evidence explain how local results, runtime observations and business outcomes fit into a truthful delivery statement.