Operations

Running a Countersign server day to day: the command-line tool, monitoring, integrity checks and what to do when one fails, backup and restore, exports for auditors, and the routine tasks. All example output on this page was produced by running the commands against scratch stores; your hashes and times will differ.

Contents

• The command-line tool
  • init
  • status
  • verify
  • config validate
  • config show
  • config check
  • config propose, approve, reject, withdraw
  • config apply
  • import
  • export
• Monitoring
• Integrity
• Backup and restore runbook
• Exporting for auditors
• Logs and clocks
• Routine tasks

The command-line tool

npm run cli -- <command> [options]        # the same as: node scripts/countersign.ts <command> [options]

The tool works directly on the store named by the same COUNTERSIGN_STORAGE, COUNTERSIGN_DATABASE_URL and COUNTERSIGN_DATA_FILE variables the server uses, through the same engine. It does not talk to a running server and does not need one. Everything it changes is an audit entry like any other. It needs Node.js 22.18 or newer.

$ npm run cli -- --help
The command-line tool for the people who run a Countersign server. It talks to the store named by the same
COUNTERSIGN_* environment variables the server uses, through the same engine, so everything it does is an audit entry
like any other. Needs Node 22.18 or newer (it runs TypeScript directly):

  node scripts/countersign.ts <command> [options]

  init      [--config <file|preset:id>] [--demo-data]   create the schema and the first configuration
  status                                                 store, head hash, configuration, integrity
  verify    [--expect-head <hash>] [--expect-entry <n>:<hash>]   re-check every entry from the first one
  config    validate <file> | show | check <file>
            propose <file> --reason … | approve [--meaning …] | reject --reason … | withdraw     (two people, two steps)
            apply <file> --reason … --reviewer-id … --reviewer-name … [--force]                   (one step)
  import    --mapping <file> --dir <folder> [--mode approved|review] [--dry-run] [--allow-partial] [--report <folder>]
  export    [--what audit|records|config] [--format json|jsonl|csv] [--out <file>]

Commands that write (config propose, approve, reject, withdraw and apply; import) need to know who is acting:
  --actor-id <id> --actor-name "<name>" --actor-roles <role,role>
They are recorded with sign-in method "cli" and the operating-system account that ran the tool.

General behaviour:

The command line asserts identity; it does not authenticate. Whoever can run the tool with database credentials decides which name is recorded. The entry therefore also records the operating-system account and host ("auth": {"method": "cli", "issuer": "jdoe@ops-01", "authTime": "…"}). Use the person's real id (the same id the identity provider gives them, so that their entries line up), restrict who can run the tool, and cover it in a procedure.

init

Creates the store's schema if needed and writes the first configuration. Does nothing if the store already has entries.

Without --config, COUNTERSIGN_CONFIG is used; without --demo-data, COUNTERSIGN_DEMO_DATA.

$ export COUNTERSIGN_STORAGE=sqlite COUNTERSIGN_DATA_FILE=./scratch/lab.sqlite
$ npm run cli -- init --config examples/config/lab-equipment.json
Initialised the sqlite store with 1 entry from examples/config/lab-equipment.json. Head #1 d6c42fbaa8ada24a95b5f5f67fdfa318f9f3b9e0c2c38482cc52794f4722945a
$ npm run cli -- init --config examples/config/lab-equipment.json
The store already holds 1 entry; nothing to do.

With demo data, on PostgreSQL:

$ npm run cli -- init --config preset:fleet --demo-data
Initialised the postgres store with 290 entries from preset:fleet. Head #290 1780d6e781351665621ddf081638c81c089441d8fa814422e8d4dc73879ff57d

Errors: Give --config <file> or --config preset:<id> (or set COUNTERSIGN_CONFIG)., --demo-data needs --config preset:<id>., Unknown preset "x". Presets: fleet, property, clinic., and The configuration <file> is not valid: followed by the list of problems.

The entry is written by the actor System with sign-in method system. Record the head hash it prints: it is the first anchor of your trail.

status

A summary of the store. Reads and replays the whole trail.

$ npm run cli -- status
store          postgres
entries        290
head hash      1780d6e781351665621ddf081638c81c089441d8fa814422e8d4dc73879ff57d
configuration  fleet "Fleet configuration", revision 1, SHA-256 547e2d8aea0405603c034b4fc1637fcc516e7ebbb59e5ad5509236947104cd8d
records        103 (104 including removed)
integrity      ok

With a fault:

$ npm run cli -- status
store          postgres
entries        290
head hash      1780d6e781351665621ddf081638c81c089441d8fa814422e8d4dc73879ff57d
configuration  fleet "Fleet configuration", revision 1, SHA-256 547e2d8aea0405603c034b4fc1637fcc516e7ebbb59e5ad5509236947104cd8d
records        93 (93 including removed)
integrity      FAULT at #256: its contents were changed after it was written (the hash no longer matches)

verify

Reads every entry from the store and checks it twice, independent of any running server: the hash chain alone, and a full replay that re-runs every command.

$ npm run cli -- verify --expect-head 1780d6e781351665621ddf081638c81c089441d8fa814422e8d4dc73879ff57d
entries read    290
hash chain      intact
replay          all 290 entries reproduce
head hash       1780d6e781351665621ddf081638c81c089441d8fa814422e8d4dc73879ff57d
expected head   matches

With an older anchor, after the trail has grown:

$ npm run cli -- verify --expect-entry 4:a86662e2aeee39237262c03cfad66181483c740446ed85bc9f5bbafcab569fff
entries read    16
hash chain      intact
replay          all 16 entries reproduce
head hash       7cdb4906fe01bb67f5e417696f80a7da76d3738b1445d2643fd21bc67741d708
anchor #4      matches

Exit code 1 if the chain is broken, replay fails, or the expected head or entry does not match:

$ npm run cli -- verify
entries read    290
hash chain      BROKEN at #256: contents changed after writing
replay          FAILED at #256: its contents were changed after it was written (the hash no longer matches)
head hash       1780d6e781351665621ddf081638c81c089441d8fa814422e8d4dc73879ff57d

$ npm run cli -- verify --expect-head 7cdb4906fe01bb67f5e417696f80a7da76d3738b1445d2643fd21bc67741d708
entries read    10
hash chain      intact
replay          all 10 entries reproduce
head hash       ec914da118232ab6129e28d84710aa4e9bae072c667329b57892f17481b4c1d9
expected head   DOES NOT MATCH 7cdb4906fe01bb67f5e417696f80a7da76d3738b1445d2643fd21bc67741d708

The second example is a trail whose newest entries were cut off: it verifies on its own, and only the expected head shows that something is missing. See Integrity.

config validate

Checks a configuration document without touching any store. Needs no environment variables.

$ npm run cli -- config validate examples/config/lab-equipment.json
examples/config/lab-equipment.json is valid: 5 record types, 8 roles. SHA-256 deedf81469275b0b0924282f745841a969e9720a4d76547620b67870aa2d3760

An invalid document lists every problem and exits with 1; see Configuration. A file that is not JSON fails with the parser's message (countersign: Expected property name or '}' in JSON at position 2 (line 2 column 1)).

config show

Prints the configuration in force as JSON, to standard output or to --out <file>.

$ npm run cli -- config show
{
  "schema": 1,
  "id": "lab-equipment",
  "name": "Laboratory equipment register",
  "description": "Instruments by site and laboratory, grouped by instrument class, with their calibration settings.",
  …

Use it to start a change from what is really in force: npm run cli -- config show --out current.json, edit a copy, validate, apply. Before init: The store has no configuration yet. Run init.

config check

Compares a configuration document with the one in force and with the records that exist. Writes nothing.

$ npm run cli -- config check org-v2.json
org-v2.json against revision 1: 2 changes, 1 reason to refuse it.
  ~ Unit › Kind: choices: drops Site
  + Position: new field Reports to (manager: reference to position)
  refused: Unit: 1 record holds a value for "Kind" that the new rules refuse (UN-0003): Kind: choose one of Legal entity, Division, Department, Team. Change those records first.

Lines with +, - and ~ are what was added, removed and changed; refused: lines are the reasons the engine would refuse the change as things stand; note: lines are consequences that are accepted (a field that becomes required, a changed order of inheritance). Exit code 1 if there is a reason to refuse. The rules are in Configuration.

config propose, approve, reject, withdraw

Changes the configuration in two steps by two people: the same propose-config and approve-config commands the application issues, so a change proposed here can be approved there and the other way round.

$ npm run cli -- config propose org-v2.json --actor-id leif --actor-name "Leif Andersson" --actor-roles system-owner \
    --reason "CR-2: reporting line"
  + Position: new field Reports to (manager: reference to position)
Proposed as entry #246 (SHA-256 a10ff33a7a445167e953da80d8041924a0070f91d752578bc3336aab4710724f). It takes effect when a second person approves it: in the application, or with "config approve".
$ npm run cli -- config approve --actor-id leif --actor-name "Leif Andersson" --actor-roles system-owner
countersign: Four eyes: you proposed this configuration change, so someone else has to approve it.
$ npm run cli -- config approve --actor-id mina --actor-name "Mina Castell" --actor-roles system-owner
Configuration revision 2 is in effect (entry #247, SHA-256 a10ff33a7a445167e953da80d8041924a0070f91d752578bc3336aab4710724f), proposed by Leif Andersson in entry #246.

The engine keeps proposer and approver apart by their ids; on the command line both ids are what the operators state. Where two people can sign in to the application, let the second step happen there: then the approver is authenticated by the identity provider and needs a recent sign-in.

config apply

Replaces the configuration in one step. One audit entry; the second person is a reviewer named by the operator. Use it for the first change on a system where nobody can sign in yet, or in an emergency; otherwise prefer the two steps above.

$ npm run cli -- config apply lab-equipment-v2.json \
    --actor-id oskar --actor-name "Oskar Lund" --actor-roles system-owner \
    --reason "CR-2026-041: add the room to instruments" \
    --reviewer-id kai --reviewer-name "Kai Tanaka"
Configuration revision 2 is in effect (entry #2, SHA-256 e707fa94555e69db28e2d35761b0702d0d48026ab14a1f1551ab5db223277e2e).

Refusals, each with exit code 1 and nothing written:

countersign: Setting the configuration needs a reason.
countersign: A configuration change needs a reviewer: a second person.
countersign: Four eyes: the reviewer must be someone other than you.
countersign: As Quality Assurance you can't change the configuration. A role needs the "configure" permission.
countersign: The configuration is unchanged.
countersign: Record type "site" has records, so its prefix ST can't change.
countersign: Field "gmp" of "site" has records, so its type can't change from boolean to text.
countersign: The change contradicts existing records (see above). Change those records first, or pass --force to apply it as the engine of version 1.0 would have.

What may and may not change when records exist, and what a change does to existing records, is in Configuration. Running servers pick the new configuration up with the next request; people see it after their next poll (within 15 seconds) or reload.

import

Brings records over from another system. The full description, with the mapping file and the reports, is in Migration.

$ npm run cli -- import --mapping examples/migration/mapping.json --dir examples/migration \
    --actor-id jdoe --actor-name "Jane Doe" --actor-roles system-owner --dry-run --report ./scratch
wrote scratch/migration-MIG-2026-11-dry-run.md
wrote scratch/migration-MIG-2026-11-dry-run.json
wrote scratch/migration-MIG-2026-11-dry-run-cross-reference.csv
class          read 2, would import 2, already present 0, refused 0
region         read 2, would import 2, already present 0, refused 0
depot          read 3, would import 2, already present 0, refused 1
vehicle        read 6, would import 3, already present 0, refused 3
definition     read 3, would import 3, already present 0, refused 0
value          read 4, would import 3, already present 0, refused 1

Exit code 1 when the rehearsal refused rows and nothing was written; 0 when rows were written (also with --allow-partial and some rows refused: read the report). Nothing is imported while the trail has an integrity fault: The audit trail has an integrity fault at #256; nothing is imported.

export

Writes data to standard output or to --out <file> (folders are created).

$ npm run cli -- export --what audit --format jsonl --out exports/trail.jsonl
wrote exports/trail.jsonl
$ npm run cli -- export --format csv | head -3
seq,at_utc,actor_id,actor_name,actor_roles,action,record_id,record_type,record_name,version,change,reason,signature_meaning,auth_method,auth_time,prev_hash,hash
1,2026-11-11T11:20:04.641Z,system,System,,configure,CONFIG,config,Laboratory equipment register,1,configuration deedf8146927… in effect,Initial configuration from examples/config/lab-equipment.json.,,system,,0000000000000000000000000000000000000000000000000000000000000000,d6c42fbaa8ada24a95b5f5f67fdfa318f9f3b9e0c2c38482cc52794f4722945a
2,2026-11-11T11:20:22.234Z,oskar,Oskar Lund,system-owner,configure,CONFIG,config,Laboratory equipment register,2,"configuration e707fa94555e… in effect, replacing deedf8146927…",CR-2026-041: add the room to instruments,,cli,2026-11-11T11:20:22.234Z,d6c42fbaa8ada24a95b5f5f67fdfa318f9f3b9e0c2c38482cc52794f4722945a,02d56b21abba91c3f290db5a026afedec4117bdd53329b9ec44512b5191f5a5b

CSV columns: seq, at_utc, actor_id, actor_name, actor_roles (space-separated), action, record_id, record_type, record_name, version, change (a one-line description), reason, signature_meaning (the approver's statement, on approve entries), auth_method, auth_time, prev_hash, hash. Lines end with CRLF. Cells that begin with =, +, - or @ are prefixed with an apostrophe so that a spreadsheet does not take them for formulas.

Things to know:

• --what other than the three values fails: --what is audit, records or config.

• --format is checked against --what: --format xml is not available for --what audit., --format csv is not available for --what records. Records and the configuration exist as JSON only.

• On a store with an integrity fault export refuses and writes nothing, with exit code 1:

  countersign: The audit trail has an integrity fault at #5 (its contents were changed after it was written (the hash no longer matches)); only the entries before it can be trusted. Nothing was exported. Run verify.
  

  To secure the damaged trail as evidence, copy it with the database's own tools (see Investigating).

• export always writes everything; there is no date or record filter. Filter the CSV, or query the reporting view.

Monitoring

The health endpoint

GET /api/health needs no sign-in and never caches. It answers 200 when all is well and 503 otherwise.

{
  "status": "ok",
  "mode": "server",
  "version": "1.0.0",
  "storage": "sqlite",
  "auth": "demo",
  "entries": 295,
  "headHash": "38459ef97bf1090ff5fddbb56d45b6b7b87372806115179ad2173ebbca38e4d5",
  "headAt": "2026-11-11T11:23:26.304Z",
  "config": {
    "id": "fleet",
    "name": "Fleet configuration",
    "revision": 1,
    "hash": "547e2d8aea0405603c034b4fc1637fcc516e7ebbb59e5ad5509236947104cd8d"
  },
  "configProposal": null,
  "fault": null
}

In demo mode the answer is {"status":"ok","mode":"demo","version":"1.0.0"}: if you see that in production, the server is running without COUNTERSIGN_STORAGE, as a demo that stores nothing. Alert on it.

Each replica answers for itself. Poll every replica, not only the load balancer.

What to alert on

A running server checks every new entry it reads and notices when the store's head is no longer the one it read. It does not re-read old entries: a change to an old entry made behind a running server's back is found by verify and at the next start, not by /api/health. That is why the scheduled verify belongs to monitoring.

Integrity

The three checks

Anchoring the head hash

The head hash commits to the whole history. An anchor is a copy of it kept where a database administrator cannot change it. On a schedule (daily is common; choose by risk):

1. Read the head: npm run cli -- status (or /api/health, or SELECT seq, hash FROM countersign_entries ORDER BY seq DESC LIMIT 1).
2. Store the entry number, hash and time outside the database and outside the reach of its administrators: a write-once object store, a signed e-mail to a records mailbox, a line in another validated system, a printed and signed log sheet. What matters is that changing it later needs different people.

Checking against an anchor:

• If the anchored entry is still the newest: npm run cli -- verify --expect-head <hash>.

• Usually the trail has grown since. Then use the anchor's entry number and hash:

  npm run cli -- verify --expect-entry 290:1780d6e781351665621ddf081638c81c089441d8fa814422e8d4dc73879ff57d
  

  The command verifies the whole chain and checks that entry 290 still has the anchored hash. Because every entry's hash depends on all entries before it, an intact chain that contains the anchored hash at the anchored number has the anchored history as its beginning.

When an integrity fault is reported

What the system does by itself:

• The affected server process keeps serving reads of the state up to the entry before the fault, and shows the fault.

• It refuses every write with HTTP 503:

  {"error":"The audit trail failed its integrity check at entry #295 (the store holds 294 entries, fewer than the 295 already read: entries were removed). Changes are refused until it has been investigated."}
  

• /api/health answers 503 with "status": "integrity-fault" and the fault; GET /api/entries returns the verified entries and the fault object.

• The state is kept until the process restarts. After a restart the process reads the store afresh: the fault reappears if the stored trail is itself inconsistent, and disappears if the store is consistent again (for example after a restore, or after a truncation that left a shorter but consistent trail, which is exactly the case an anchor is for).

The fault messages:

Investigating

Treat it as a potential data-integrity incident under your procedures. Do not restart servers first: a restart can make the evidence of a truncation disappear from /api/health.

1. Preserve. Save the /api/health output of every replica and the application logs. Take a database backup or snapshot as it is now, labelled as evidence.
2. Stop changes. The servers already refuse writes. Leave them running for reading, or take the application offline if the data shown may be wrong.
3. Locate. npm run cli -- verify names the first failing entry and how it fails. status shows the state up to it.
4. Compare with what you know to be good:
   • the anchors: which is the last anchored head that is still in the trail?
   • the last good backup: restore it into a separate database and verify it; export both trails as JSONL and diff them from the failing entry on (export refuses on the faulty store, so take its entries with the database's tools: psql -Atc "SELECT entry FROM countersign_entries ORDER BY seq", or copy the JSONL file);
   • inserted_at in PostgreSQL against at: rows written at an unusual time;
   • database logs and audit logs: who connected with which role, were triggers disabled (ALTER TABLE … DISABLE TRIGGER), was there a restore or failover?
5. Decide the cause: infrastructure (a restore, a failover, disk damage), an operational mistake, a software defect (an upgrade whose engine replays old entries differently: roll the application back, the data is intact), or deliberate manipulation.
6. Recover. The only way to repair a trail is to replace the store's content with a verified good copy; entries cannot be edited back into shape, and Countersign has no repair command. Restore the last backup that verifies (next section), check it against the anchor, then restart the servers.
7. Account for the gap. Entries written after the restored backup are lost from the system. Re-enter them through the application as new, attributable actions, with reasons that refer to the incident. The preserved copy of the faulty database remains the record of what was there.
8. Document in your deviation system: what was found, cause, what was restored, what was re-entered, the new anchor.

Backup and restore runbook

The whole system is one table (or one file). What you need in order to rebuild an installation:

The configuration document needs no separate backup: it is in the trail (config show).

Backup

The JSONL export is worth keeping next to database backups: it is plain text, can be verified with twenty lines of code and no Countersign (Architecture), and can be loaded into any store.

Restore

1. Stop all servers (or scale to zero).

2. Create a new, empty database and restore into it; see Storage for the commands. Do not restore over the damaged one: keep it.

3. Verify, with the head noted at backup or an anchor:

   $ npm run cli -- verify --expect-head 1780d6e781351665621ddf081638c81c089441d8fa814422e8d4dc73879ff57d
   entries read    290
   hash chain      intact
   replay          all 290 entries reproduce
   head hash       1780d6e781351665621ddf081638c81c089441d8fa814422e8d4dc73879ff57d
   expected head   matches
   

4. Check the triggers and the application role's privileges (PostgreSQL).

5. Point the servers at the restored database and start them. /api/health must show ok with the expected entries and headHash on every replica.

6. Determine what was written after the backup and is now missing (from the last anchor, monitoring history of entries, or the damaged database's export) and have it re-entered.

7. Anchor the new head. File the outputs.

Rehearse this at least once before go-live and regularly afterwards, and keep the rehearsal's output: a backup is only proven by a verified restore.

Exporting for auditors

A package that stands on its own:

npm run cli -- verify > verify.txt && \
npm run cli -- export --what audit --format jsonl --out package/audit-trail.jsonl && \
npm run cli -- export --what audit --format csv   --out package/audit-trail.csv && \
npm run cli -- export --what records              --out package/records.json && \
npm run cli -- export --what config               --out package/configuration.json && \
npm run cli -- status > package/status.txt && cp verify.txt package/ && \
( cd package && sha256sum * > SHA256SUMS )

verify comes first so that its output is part of the package; export itself refuses on a store with an integrity fault, and && stops the sequence at the first failure.

People with access to the application can also export from the browser: the audit trail page offers CSV and JSON downloads, and the read-only Configuration page shows record types, fields, inheritance chains, roles and the configuration's history. The command-line exports are the ones to use for formal evidence, because they come with verify.

The CSV is written for people: times in UTC, one line of description per entry. It does not contain the data before and after an edit; the JSON formats do. Remember that an export contains everything every record holds, including personal data: handle it accordingly.

Logs and clocks

Logs

• The audit trail is not a log file, and log files are not the audit trail. Everything that changes a record is in the trail. What is not in the trail: sign-ins and sign-outs, failed sign-ins, refused commands, and reads.
• Countersign writes very little to its own output. Refused commands are answered to the client (HTTP 422 with the reason) and not logged. The reason for a store-unreachable health status is logged (countersign: the store could not be reached: …) and deliberately not returned to the caller.
• Sign-in events are recorded by the identity provider. Requests, including refused ones and reads, are recorded by the reverse proxy's access log. Keep both for as long as your procedures require; they are the source for questions such as "who looked at this" and "were there attempts".
• Database logs record connections and, if you enable it, statements by privileged roles. They are the evidence when an integrity fault is investigated. Enable connection logging and logging of DDL at least.
• Do not log request bodies at the proxy unless you mean to: they contain record data.
• Secrets never appear in responses. The session cookie and the proxy secret appear in request headers; exclude those headers from logs.

Clocks

• The time of an entry (at) is the clock of the application server that handled the command, in UTC with milliseconds. The browser's clock is never used.

• Synchronise every application server with a reliable time source and monitor the offset. The time in the audit trail is only as trustworthy as that.

• Entry times never go backwards. If a server's clock is behind the newest entry (two servers that differ by a moment, or a clock stepped back), the new entry gets the newest entry's time instead of an earlier one. A clock that is far behind therefore produces a run of entries with identical times: a sign of a clock problem.

• A clock that is far ahead is worse: its entries carry future times, and every later entry from any server is held at that future time until real time catches up. Alert on offset.

• PostgreSQL's inserted_at column is the database server's clock at insert, independent of the application. A large difference between at and inserted_at points to a clock problem on one side:

  SELECT seq, at, inserted_at, inserted_at - at AS lag FROM countersign_entries
  WHERE abs(extract(epoch FROM inserted_at - at)) > 5 ORDER BY seq;
  

  (Demo data and migrated trails legitimately show large differences: their entries carry their original times.)

• Dates in fields (date) are calendar dates without a time zone. The user interface shows entry times in the viewer's local time zone; exports are in UTC.

Routine tasks