Standard operating procedure templates

How to use these templates

Ten short procedures follow. Each is a starting point: copy it into your own SOP format, give it your document number, replace the blanks and role names, delete what does not apply, and approve it under your document control. The technical steps and expected outputs are those of Countersign 1.0.0; the organisational choices (who, how often, which records) are yours. countersign stands for the command-line tool (npm run cli -- in the standalone repository), run with the installation's COUNTERSIGN_* environment. Operator detail is in Operations.

Roles used below: process owner, system owner, QA, system administrator, database administrator (DBA), identity administrator, reviewer, user. See the validation plan, section 4.

SOP 1: User access and role management

Purpose. Ensure that only authorised, trained people can use Countersign, each with the roles their function needs, and that access is withdrawn promptly.

Scope. All access to the application. Roles are not managed inside Countersign: a person's roles are the groups or application roles they hold in the identity provider, translated by COUNTERSIGN_ROLE_MAP to role ids of the configuration. Access to the server, the database and the command-line tool is covered by ______ (your privileged access procedure).

Responsibilities.

Procedure.

1. Request: the line manager submits the access form naming the person, the role and the reason.
2. Check: the process owner verifies training is complete, the role is in the approved matrix, the person holds no conflicting privilege (database, server or identity administration), and the person has one personal account.
3. Grant: the identity administrator adds the person to the group for that role and records date and time.
4. Verify: the person signs in; the application shows the expected name and role. A role change takes effect at the person's next sign-in.
5. Change of function: treat as withdrawal plus a new request.
6. Withdrawal: on leaving or changing function, the identity administrator removes the group membership and disables the account on the effective date. A session already running stays valid until it ends, at most COUNTERSIGN_SESSION_HOURS (____ hours here). If access must end at once, the system owner rotates COUNTERSIGN_SESSION_SECRET on all servers and restarts them; this signs everybody out.
7. Shared, generic and service accounts must never be given a role.
8. Access review every ____ months: QA compares the members of each group with the access forms, and compares the roles that appear in the audit trail for the period (actor_roles in the CSV export) with what each person was entitled to. Differences are deviations.
9. Role map changes (COUNTERSIGN_ROLE_MAP) follow SOP 4, section "other controlled settings".

Records. Access forms; group membership lists at each review; access review report; training records.

SOP 2: Audit trail review

Purpose. Detect unauthorised, unexplained or careless changes and confirm the integrity of the audit trail.

Scope. The complete audit trail of the production installation. Two kinds of review: (a) review of a record's own history by the approver, as part of every approval; (b) periodic review of the whole trail, described here.

Responsibilities. Reviewer: a trained person independent of the changes reviewed, with read access (any signed-in account) and no need for a write role. System owner: provides verify output and logs. QA: reviews the report.

Procedure.

1. Frequency: every ____ (by risk; monthly is common for active master data), and additionally after a migration, a configuration change, an upgrade, a restore and an integrity incident.
2. Determine the period: from the entry after the last one reviewed to the current head.
3. Obtain from the system owner: output of countersign verify; the anchor comparison (SOP 5); the health monitoring history; the web-server log extract for refused requests.
4. Export the period: audit trail as CSV for working and as JSON Lines for the file (countersign export --what audit --format jsonl --out <file>).
5. Work through the checklist in Data integrity, section 9: integrity checks 1 to 7, content checks 8 to 23.
6. For every finding open a deviation or incident record. Do not correct anything in the application as part of the review; corrections go through the normal workflow.
7. Take a new anchor (SOP 5) and note it in the report.
8. Sign the checklist; QA reviews and signs.

Records. Completed checklist; exports of the period; verify output; findings and their record numbers.

SOP 3: Backup and restore

Purpose. Ensure the audit trail, which is the only stored data, can be recovered completely and that a restore is proven complete before the system is used again.

Scope. The database (table countersign_entries), the configuration of the installation (environment, role map), and the anchor register. A restore is the one routine operation that can silently remove entries: a database restored from an older backup is a shorter but perfectly consistent trail.

Responsibilities. DBA: backups and restores. System owner: decides on a restore, verifies it, releases the system. QA: approves the release after a restore of production.

Procedure: backup.

1. Back up the database every ____ with point-in-time recovery (continuous archiving), retained for ____.
2. Store backups where the application's account cannot reach them, at a second location.
3. After each scheduled full backup, record the head (number of entries and head hash from countersign status) with the backup's identifier. This is an anchor (SOP 5).
4. Monitor backup jobs; a failed backup is an incident.
5. Back up the environment definition and role map with your configuration management; secrets in the secret store.

Procedure: restore test (every ____ months, and in IQ).

1. Restore the latest backup into a separate database.
2. Point the tool at it and run countersign verify --expect-head <head hash recorded for that backup>.
3. Expected: hash chain intact, replay all <n> entries reproduce, expected head matches. Record the result.

Procedure: restore of production.

1. Open an incident. Stop all application servers.
2. Before restoring, if the damaged database is still readable, export what it holds: countersign export --what audit --format jsonl --out before-restore.jsonl, and note countersign status.
3. Restore to the latest possible point in time.
4. Run countersign verify. It must report the chain intact and all entries reproduced.
5. Compare with the newest anchor: countersign verify --expect-entry <N>:<H> must print anchor #<N> matches (Data integrity, section 7.2). If the restored head is older than the newest anchor, entries were lost.
6. Determine the lost entries: everything after the restored head, from before-restore.jsonl, the last periodic export, or the anchor register. List them (who, what record, what action).
7. Lost changes are not re-inserted into the database. The people concerned repeat them through the application, with a reason referring to the incident. The times will be the new times; the incident record explains the difference.
8. Confirm triggers and privileges (IQ steps 6.2, 6.9, 6.10), start the servers, check /api/health.
9. Take a new anchor. QA releases the system.

Records. Backup logs; restore test records; incident record with the list of lost entries and their re-entry.

SOP 4: Configuration change control

Purpose. Ensure that changes to the configuration document (record types, fields, roles, permissions) are assessed, reviewed, tested, applied by an authorised person and traceable.

Scope. The configuration document applied with countersign config apply. The same control applies to "other controlled settings": COUNTERSIGN_ROLE_MAP, the sign-in and session variables, and database roles and grants.

Background: the application records a configuration change as an audit entry with the operator's stated identity, the reason, the named reviewer and the document's SHA-256. The engine requires the configure permission, a reason and a reviewer whose id differs from the operator's. It cannot verify that the reviewer really reviewed, or that the operator is who the command line says. This procedure supplies that assurance.

Responsibilities.

Procedure.

1. Change request: describe the change, the reason, the impact on existing records, roles, training, SOPs and validation documents, and the risk.
2. Prepare the new document from the current one (countersign config show --out current.json). Keep both under version control.
3. Validate: countersign config validate new.json. Record the SHA-256 it prints.
4. Review: the reviewer compares current.json and new.json line by line against the change request, and signs the change record, which states the SHA-256 of step 3.
5. Test: apply in the test environment (step 7) and execute the tests QA decided (at least: the changed function, one unchanged function, countersign verify).
6. Approve: process owner and QA approve the change for production.
7. Apply, with a witness, on the production host: countersign config apply new.json --actor-id <operator's id> --actor-name "<operator's name>" --actor-roles <role with configure> --reviewer-id <reviewer's id> --reviewer-name "<reviewer's name>" --reason "<change record number and summary>". Use the identifiers the identity provider uses for these people, so that the trail is consistent.
8. Expected output: Configuration revision <r> is in effect (entry #<n>, SHA-256 <hash>). The hash must equal step 3. Typical refusals: "A configuration change needs a reviewer: a second person."; "Four eyes: the reviewer must be someone other than you."; a message that a record type still has records, or that a field's type cannot change.
9. Verify: countersign status shows the new revision and hash; /api/health shows the same; the changed function works in the application.
10. Take an anchor (SOP 5). Close the change record with entry number and hash. Update training and documents.

A configuration cannot be rolled back by deletion: to undo a change, apply the previous document as a new change.

Records. Change record with both documents, the SHA-256, review signature, test evidence, tool output, entry number.

SOP 5: Head-hash anchoring

Purpose. Keep an independent reference that makes removal of entries, rollback to an old backup or replacement of the audit trail evident.

Scope. The production audit trail. Background and commands: Data integrity, section 7.

Responsibilities. System owner: takes anchors. Anchor custodian (not a DBA, no write access to the database): keeps the register. Reviewer: verifies anchors in the audit trail review.

Procedure.

1. When: every ____ (daily or weekly by risk); with each full backup; before and after a restore, a migration, an upgrade and a configuration change; at each audit trail review.
2. Take: run countersign status; confirm integrity ok; note the number of entries N and the head hash H.
3. Verify the previous anchor first: countersign verify --expect-entry <previous N>:<previous H> must print anchor #<N> matches and exit with 0. A failure is an integrity incident (SOP 6). At each audit trail review also run the standalone script on an export, as the method that does not depend on the application.
4. Record date and time, N, H, configuration SHA-256 and your name in the anchor register and send it to the custodian by ______ (the channel that the database administrators cannot alter).
5. The custodian files it and confirms receipt.

Records. Anchor register (Data integrity, section 7.3).

SOP 6: Integrity-fault handling

Purpose. Respond correctly when the system reports that its audit trail failed verification.

Scope. Any of: /api/health answering HTTP 503 with "status":"integrity-fault"; users receiving "The audit trail failed its integrity check at entry #n (…). Changes are refused until it has been investigated."; countersign status printing integrity FAULT; countersign verify exiting with 1; a failed anchor comparison.

What the system does on its own: it refuses every change, keeps serving the state before the faulty entry for reading, and keeps the fault until the server process is restarted against a store that verifies. It repairs nothing, and there is no repair tool.

Responsibilities. System owner: leads. DBA: preserves and analyses the database. QA: classifies and approves the resolution. Security function: involved if manipulation is suspected.

Procedure.

1. Record the time, the entry number and the message from /api/health. Open a critical incident. Inform users that changes are suspended; reading remains possible but shows the state before the fault. Start SOP 9.

2. Do not restart the servers yet and do not change anything in the database. Never disable triggers or edit rows to "fix" the trail.

3. Preserve evidence: a full copy of the database (dump or snapshot); database logs; countersign verify and countersign status output; the web-server and identity-provider logs for the period; the anchor register.

4. Identify the kind of fault from the message:

   Message contains	Meaning	First question
   "its contents were changed after it was written"	The stored entry differs from what was hashed	Who had write access outside the application? Storage corruption?
   "the chain is broken"	An entry does not follow its predecessor	Was an entry removed, inserted or replaced?
   "it breaks a rule: …"	The entry could not have been produced by the application under the rules, or the installed software judges it differently	Was it inserted directly? Was the software version changed (SOP 10)?
   "re-running its command gives a different result"	As above	As above
   "the entry is not readable"	The stored text is not an entry	Direct insert or corruption
   "entries were removed"	The store holds fewer entries than the running server had read	Restore, failover to a stale replica, or deletion
   "the trail was rewritten"	The newest entry changed under the running server	As above

5. Determine the extent: the first faulty entry and everything after it are suspect. Compare with the newest anchor and the last periodic export to establish the last state known to be good.

6. Determine the cause with the DBA: database audit log (DDL, disabled triggers, sessions of privileged accounts), infrastructure events (failover, restore, storage errors), recent upgrades.

7. Resolve. The only supported recovery is to bring the store back to a state that verifies: restore under SOP 3 to a point before the damage, then verify against the anchor. If the cause was a software upgrade judging old entries differently, redeploy the previous version (SOP 10) and report the defect to the project.

8. Assess impact: which records and decisions relied on entries that were lost or suspect; which changes must be repeated through the application.

9. Before release: countersign verify passes; anchor comparison passes; IQ steps 6.2, 6.9 and 6.10 pass; QA approves. Start the servers; check /api/health is ok; take a new anchor.

10. Close the incident with root cause and corrective and preventive actions. If manipulation is confirmed, follow your data integrity breach procedure, including regulatory notification where required.

Records. Incident record; preserved evidence; verification outputs; impact assessment; release approval.

SOP 7: Migration

Purpose. Bring records over from another system completely, correctly and traceably.

Scope. Use of countersign import. Technical detail: Migration.

Responsibilities. Process owner: owns the data and signs the reconciliation. System owner: runs the tool (role with the import permission). Second person: checks the reconciliation. QA: approves plan and report.

Procedure.

1. Plan: source system, record types, volumes, batch identifier (use the change record number), mode, acceptance criteria, sampling plan. Choose the mode deliberately: approved brings records in as approved versions marked with their origin, without an approver in this system, and is justified only when the source's approval can be relied on; review creates and confirms them so that approvers sign each one here.
2. Extract the source files; record their SHA-256 and row counts; keep them unchanged and retained.
3. Write the mapping file; review it against the configuration.
4. Rehearse in the test environment with a full extract: countersign import --mapping <file> --dir <folder> --dry-run --report <folder> --actor-id … --actor-name … --actor-roles …. Review every refused row in the report; correct the source or the mapping; repeat until the result is accepted.
5. Take an anchor (SOP 5). Run the import in production without --dry-run. The tool rehearses again first and writes nothing if any row is refused, unless --allow-partial is given; use that option only if the plan allows partial import and names what happens to refused rows.
6. Keep the three outputs: the Markdown report (source hashes, totals, refused rows, sign-off table), the JSON report and the cross-reference CSV (old id to new id).
7. Reconcile: for every record type, rows read = imported + already present + refused; the source hashes in the report equal step 2; the entry range in the report matches the audit trail.
8. Sample: compare ____ % of records, at least ____ per type, field by field against the source, including references and option values.
9. In review mode: approvers approve the records in the application before they are used.
10. Run countersign verify; take an anchor; sign the report's sign-off table (performed by, checked by, QA).

A second run of the same batch imports only what is not there yet; a source record cannot be imported twice.

Records. Migration plan; source files and hashes; mapping; reports; cross-reference; sample comparison; sign-off.

SOP 8: Periodic review

Purpose. Confirm at defined intervals that the system remains in a validated state and fit for its use.

Scope. The production installation, its documents and procedures. Interval: ____ months.

Responsibilities. System owner prepares; process owner and QA review and approve.

Procedure. Review and record each item:

Conclude: validated state confirmed, actions required, or revalidation required. Set the date of the next review.

Records. Periodic review report with evidence and action list.

SOP 9: Business continuity

Purpose. Keep the supported process running when Countersign is unavailable or refuses changes, and bring the system back into a complete state afterwards.

Scope. Unavailability of the application, the database or the identity provider, and integrity faults (in which reading still works but changes are refused).

Responsibilities. Process owner: decides to invoke the fallback. System owner: restores service. QA: reviews the catch-up.

Procedure.

1. Criticality (complete): maximum tolerable outage ____; maximum tolerable data loss ____.
2. Reading during an outage: the last periodic export of the records (countersign export --what records --out <file>, taken every ____) is the reference copy. State where it is kept and who may use it.
3. Changes during an outage: record each change on the fallback form with the same information the system would hold: record, old value, new value, reason, author with date and time, second person's review with date and time.
4. No change takes effect operationally on the fallback form without the second person's signature.
5. Restoring service: SOP 3 or SOP 6 as applicable.
6. Catch-up: the original author enters each fallback change in the application, with a reason that cites the fallback form number and the original date; the original reviewer approves. The entry times will be the catch-up times; the form is retained as the contemporaneous record.
7. QA reconciles forms against entries and closes the event.
8. Test this procedure every ____ months as a table-top or live exercise.

Records. Fallback forms; reconciliation; exercise reports.

SOP 10: Upgrade of the software

Purpose. Install a new version of Countersign, or of the platform under it, without losing the validated state.

Scope. New releases of Countersign; major versions of Node.js and PostgreSQL. Background: on start the server re-runs every stored command with the installed version of the rules engine. A new version that judged an old entry differently would report an integrity fault on a trail nobody altered. This must be found before production.

Responsibilities. System owner: plans and performs. QA: approves the change and the release. Testers: execute.

Procedure.

1. Assess: read CHANGELOG.md for every version between the installed one and the target; list changes to the rules engine, the hash computation, storage, the schema, sign-in and settings; review the project's security advisories. Record impact and risk in a change record; decide the test scope.
2. Obtain the release from the project's repository; record tag and commit; verify the dependency lock file is the release's.
3. Compatibility with your history: restore a current copy of production into a scratch database and run the new version's countersign verify against it. It must report the chain intact and all entries reproduced, with the same head hash as production. If it does not, stop: do not upgrade, and report it to the project.
4. Test environment: install; execute the IQ version and settings steps; execute OQ part A completely (npm run oq) and attach the report; regenerate and compare the traceability matrix (npm run trace); execute parts B and C and PQ scenarios according to the test scope.
5. Review this validation pack against the new version and update the affected documents.
6. Approve for production (process owner, QA).
7. Production: take an anchor (SOP 5); back up; stop the servers; install; if the release changes the database schema, the DBA applies the reviewed db/postgres.sql as the owner role; start one server; check /api/health (ok, new version, same headHash as the anchor); run countersign verify --expect-head <anchor>; start the remaining servers.
8. Repeat the IQ steps for versions, settings, triggers and privileges. Take a new anchor.
9. Roll back if any check fails: redeploy the previous version (keep it deployable), verify again, investigate.
10. Close the change record; update the system inventory and the summary of validated versions; train users if behaviour changed.

Records. Change record; verify output from step 3; OQ execution report; IQ repeat; release approval.