Deployment

How to run Countersign in production: every environment variable, Docker and docker compose, a reverse proxy in front, a Kubernetes sketch, a hardening checklist, sizing and upgrades. For the database see Storage, for sign-in Authentication, for running it day to day Operations.

Contents

• What a production installation consists of
• Environment variables
• Start-up checks
• Docker
• docker compose
• Behind a reverse proxy
• Kubernetes sketch
• Hardening checklist
• Sizing
• Upgrade procedure

What a production installation consists of

flowchart LR
  accTitle: A production installation: reverse proxy, Countersign, PostgreSQL and the identity provider
  people(["People"]) -- "HTTPS" --> proxy["Reverse proxy<br/>TLS, headers, rate limits"]
  proxy -- "HTTP" --> app["Countersign<br/>1..n Node.js processes"]
  app --> db[("PostgreSQL<br/>one table, append-only")]
  app -- "OIDC, outbound HTTPS" --> idp["Identity provider"]
  ops["Operations host<br/>command-line tool"] --> db

Environment variables

All settings come from the environment. Values are trimmed. Nothing is read from files except the configuration document and the PostgreSQL schema.

Mode and storage

Sign-in

Sessions, signatures and address

Development and tests

Invalid values stop the server with a message, for example COUNTERSIGN_STORAGE must be one of memory, file, sqlite, postgres (or unset for the demo)., COUNTERSIGN_AUTH must be one of entra, oidc, proxy, demo., COUNTERSIGN_ROLE_MAP must be JSON such as {"SG-Fleet-QA":"qa"}., COUNTERSIGN_DEMO_USERS must be JSON such as [{"id":"ann","name":"Ann Lee","roles":["qa"]}]. or COUNTERSIGN_SESSION_HOURS must be a number.

A minimal production environment

COUNTERSIGN_STORAGE=postgres
COUNTERSIGN_DATABASE_URL=postgres://countersign_app:<password>@db.internal:5432/countersign?sslmode=verify-full&sslrootcert=/etc/ssl/db-ca.pem
COUNTERSIGN_SKIP_MIGRATE=1
COUNTERSIGN_AUTH=entra
COUNTERSIGN_ENTRA_TENANT_ID=<tenant id>
COUNTERSIGN_ENTRA_CLIENT_ID=<client id>
COUNTERSIGN_ENTRA_CLIENT_SECRET=<client secret>
COUNTERSIGN_SESSION_SECRET=<openssl rand -base64 48>
COUNTERSIGN_PUBLIC_URL=https://records.example.com

The configuration is written once, before the first start, with init (see Operations), so COUNTERSIGN_CONFIG is not needed here.

Start-up checks

In server mode the application refuses to serve until its settings are complete. Every endpoint then answers HTTP 503 and lists what is missing; /api/health is the place to look:

{
  "status": "misconfigured",
  "mode": "server",
  "version": "1.0.0",
  "problems": [
    "COUNTERSIGN_DATABASE_URL is required for PostgreSQL.",
    "COUNTERSIGN_SESSION_SECRET must be at least 32 characters (for example: openssl rand -base64 48).",
    "COUNTERSIGN_ENTRA_TENANT_ID is required.",
    "COUNTERSIGN_ENTRA_CLIENT_ID is required.",
    "COUNTERSIGN_ENTRA_CLIENT_SECRET is required."
  ]
}

The store is opened on the first request, not when the process starts: a wrong database address shows as "status": "store-unreachable" on /api/health, not as a crash. Check /api/health after every deployment; see Operations.

Docker

The repository's Dockerfile builds the application (Next.js standalone output on node:22-alpine) and also contains the command-line tool, db/postgres.sql, the examples and the documents. The container runs as the unprivileged user countersign, listens on port 3000, has a writable /app/data for the SQLite and file stores, and declares a health check on /api/health. Build and run:

docker build -t countersign:1.0.0 .

docker run -d --name countersign -p 3000:3000 --env-file ./production.env countersign:1.0.0
curl -s http://localhost:3000/api/health

The command-line tool in the same image, against the same store:

docker run --rm --env-file ./production.env -v "$PWD/config:/config:ro" countersign:1.0.0 \
  node scripts/countersign.ts init --config /config/lab-equipment.json
docker run --rm --env-file ./production.env countersign:1.0.0 node scripts/countersign.ts status

Notes:

• Pin the image by version or digest and record it in your installation qualification (validation pack).
• With COUNTERSIGN_STORAGE=sqlite or file, mount a volume at the folder of COUNTERSIGN_DATA_FILE (by default /app/data), writable for the container's user; without one the trail is lost with the container.
• Pass secrets with your platform's secret mechanism, not baked into the image.
• If the PostgreSQL schema file is not found inside the image, set COUNTERSIGN_SCHEMA_FILE to its path or create the schema by hand and set COUNTERSIGN_SKIP_MIGRATE=1.

(Docker was not available on the machine this page was written on; the commands above are the standard ones for the image and were not executed there.)

docker compose

docker compose up is meant for evaluation: it starts PostgreSQL 16 and the application with

COUNTERSIGN_STORAGE=postgres
COUNTERSIGN_AUTH=demo
COUNTERSIGN_CONFIG=preset:fleet
COUNTERSIGN_DEMO_DATA=1

that is, a shared database filled with the fleet demo history, and a sign-in page where you pick one of the demo people. It also sets a fixed, published COUNTERSIGN_SESSION_SECRET and a database password of countersign, which is one more reason not to use it as it is. Open http://localhost:3000. Everything about the server path can be tried this way: several browsers as different people, approvals, the audit trail in PostgreSQL (countersign_audit_trail), the CLI.

To turn it into a real installation:

1. Take the variables from .env.example, where the Entra and OIDC variables are listed as comments, fill them in, and generate your own session secret.
2. Set COUNTERSIGN_AUTH=entra (or remove the line), remove COUNTERSIGN_DEMO_DATA, and replace COUNTERSIGN_CONFIG=preset:fleet with your own configuration file or run init yourself.
3. Start from an empty database. A store that was filled with demo data keeps it for ever; there is no way to delete entries.
4. Give PostgreSQL a real password, a volume that is backed up, and the two roles of Storage.
5. Put a TLS-terminating proxy in front.

Behind a reverse proxy

Countersign should not face the internet directly. The proxy terminates TLS; the application listens on HTTP in a private network.

What Countersign needs from the proxy

The public address is used for three things: the OIDC redirect URI (<public URL>/auth/callback), the origin check on writes (a request whose Origin header differs is refused with The request comes from another site.), and the Secure flag of the cookies (set when the address starts with https://).

Prefer COUNTERSIGN_PUBLIC_URL. If you rely on forwarded headers instead, the proxy must overwrite X-Forwarded-Proto and X-Forwarded-Host on every request, because a client that could set them could influence the origin check and the redirect URI.

nginx example

limit_req_zone $binary_remote_addr zone=cs_auth:10m  rate=10r/m;
limit_req_zone $binary_remote_addr zone=cs_write:10m rate=120r/m;

server {
  listen 80;
  server_name records.example.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;
  http2 on;
  server_name records.example.com;

  ssl_certificate     /etc/ssl/records.example.com.fullchain.pem;
  ssl_certificate_key /etc/ssl/records.example.com.key;
  ssl_protocols       TLSv1.2 TLSv1.3;

  add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
  add_header X-Content-Type-Options    "nosniff" always;
  add_header X-Frame-Options           "DENY" always;
  add_header Referrer-Policy           "same-origin" always;
  add_header Permissions-Policy        "camera=(), microphone=(), geolocation=()" always;
  add_header Content-Security-Policy   "frame-ancestors 'none'" always;

  client_max_body_size 300k;

  proxy_set_header Host              $host;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-Host  $host;
  proxy_set_header X-Forwarded-For   $remote_addr;

  location /auth/ {
    limit_req zone=cs_auth burst=10 nodelay;
    proxy_pass http://127.0.0.1:3000;
  }
  location = /api/act {
    limit_req zone=cs_write burst=30 nodelay;
    proxy_pass http://127.0.0.1:3000;
  }
  location = /api/health {
    allow 10.0.0.0/8;       # monitoring network
    deny  all;
    proxy_pass http://127.0.0.1:3000;
  }
  location / {
    proxy_pass http://127.0.0.1:3000;
  }
}

Security headers

Countersign sets Cache-Control: no-store on its API and sign-in responses and the cookie attributes described in Authentication. The Next.js application (next.config.ts) also sends X-Content-Type-Options: nosniff, X-Frame-Options: DENY and Referrer-Policy: same-origin on every response; the plain API server (scripts/api-server.ts) does not. Setting them at the proxy as well does no harm and covers both. The rest is the proxy's job:

Rate limiting

Countersign has no rate limiting of its own. Limit at the proxy:

Also limit the request body (the application refuses bodies over 256 KiB to /api/act itself) and restrict /api/health and, if you wish, /api/bootstrap to where they are needed: both answer without sign-in. /api/health reveals the entry count, the head hash and the configuration's name; /api/bootstrap the storage kind and sign-in method.

Kubernetes sketch

A starting point, not a tested manifest. Two replicas work because sessions are signed cookies and concurrent writes are settled by the database (see Architecture).

apiVersion: v1
kind: Secret
metadata:
  name: countersign
type: Opaque
stringData:
  COUNTERSIGN_DATABASE_URL: postgres://countersign_app:<password>@postgres.db.svc:5432/countersign?sslmode=verify-full
  COUNTERSIGN_ENTRA_CLIENT_SECRET: <client secret>
  COUNTERSIGN_SESSION_SECRET: <at least 32 random characters, the same for all replicas>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: countersign
spec:
  replicas: 2
  selector:
    matchLabels: { app: countersign }
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }
  template:
    metadata:
      labels: { app: countersign }
    spec:
      securityContext:
        runAsNonRoot: true
      containers:
        - name: countersign
          image: registry.example.com/countersign:1.0.0
          ports:
            - containerPort: 3000
          envFrom:
            - secretRef: { name: countersign }
          env:
            - { name: COUNTERSIGN_STORAGE, value: postgres }
            - { name: COUNTERSIGN_SKIP_MIGRATE, value: "1" }
            - { name: COUNTERSIGN_AUTH, value: entra }
            - { name: COUNTERSIGN_ENTRA_TENANT_ID, value: "<tenant id>" }
            - { name: COUNTERSIGN_ENTRA_CLIENT_ID, value: "<client id>" }
            - { name: COUNTERSIGN_PUBLIC_URL, value: "https://records.example.com" }
          startupProbe:
            httpGet: { path: /api/health, port: 3000 }
            periodSeconds: 5
            failureThreshold: 60
          readinessProbe:
            httpGet: { path: /api/health, port: 3000 }
            periodSeconds: 10
            timeoutSeconds: 5
          livenessProbe:
            httpGet: { path: /api/health, port: 3000 }
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 5
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 1Gi }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
---
apiVersion: v1
kind: Service
metadata:
  name: countersign
spec:
  selector: { app: countersign }
  ports:
    - port: 80
      targetPort: 3000

Things to decide:

• Probes. /api/health answers 200 when the store is reachable and the trail verifies, and 503 otherwise (misconfigured, store unreachable, integrity fault). As a readiness probe that is right: a replica that cannot reach the database gets no traffic. As a liveness probe it restarts replicas during a database outage and on an integrity fault, where a restart does not help; a restart loop then hides the real problem. Either give liveness a high failure threshold, as above, or use a TCP probe for liveness and keep the HTTP check for readiness and alerting.
• Start-up time. The first health check triggers the reading and replay of the whole trail. That takes seconds even for a long trail (see Sizing); the startupProbe above allows five minutes, which also covers a slow database.
• Read-only root file system. Next.js may want a writable cache directory; mount an emptyDir where your image needs one.
• The CLI runs as a Job or with kubectl exec in a pod of the same image and Secret: kubectl exec deploy/countersign -- node scripts/countersign.ts status.
• Ingress must provide what the reverse-proxy section lists.
• The schema is created once by a DBA or a Job that connects as the owner role; the Deployment uses the application role with COUNTERSIGN_SKIP_MIGRATE=1.

Hardening checklist

Sizing

What costs time and memory:

• Replay on start. Each server process reads the whole trail on its first request and replays it through the engine. Afterwards it reads only new entries.
• State in memory. Every process keeps the complete state: all entries, all records with all versions.
• The browser receives the whole trail on first load and replays it too.
• Each single command on a running server copies part of the state, so its cost grows slowly with the size of the store.

Replay works in place on an indexed copy of the state, and its time grows in proportion to the number of entries. Measurements on a development machine (Node 22, one core), with a synthetic fleet trail (create, confirm and approve of vehicles with unique plates):

The maintainers measured 7.3 s for 100,000 entries (33,000 records) with the same kind of trail. While that 30,010-entry trail was being built, one command at a time, a command took 5 ms on average.

In practice:

• Server start is not a concern up to tens of thousands of entries: a few seconds at most, plus the time to read the entries from the store.
• Memory is what to plan for: every server process holds the whole state, roughly 3 MB of heap per thousand entries in the measurement above, and so does every open browser tab.
• The browser's first load downloads the whole trail and replays it. At 30,000 entries that is about 15 MB of JSON before compression; a slow connection or an old device will feel it long before the server does. Compress responses at the proxy.
• Beyond that range, measure with a copy of your own data, on the server and in the oldest browser you support. The quickest server-side measurement is time npm run cli -- verify, which does the same replay.

There are no snapshots and no archiving in version 1.0.0, so a trail only grows. Estimate entries as records × about three (create, confirm, approve) plus three per later change, plus option values.

Other sizing notes:

Upgrade procedure

Countersign's data format is the audit entry and the rule "replaying old entries gives the same result". An upgrade must preserve both, so the procedure is built around verify.

1. Read the changelog (CHANGELOG.md) for the versions you skip: changed variables, a changed db/postgres.sql, anything about replay compatibility.

2. Assess the change under your change control and decide what to re-test (see the validation pack).

3. Back up the database and note status (entry count, head hash).

4. Rehearse on a copy. Restore the backup into a test database and run the new version's CLI against it:

   npm run cli -- verify --expect-head <head hash from step 3>
   

   replay all N entries reproduce means the new engine gives the same result for every entry of your history. If replay fails, stop: do not deploy that version on your data.

5. Run the new version's tests (npm test, or npm run oq -- --out report.md for the record) in the test environment, and click through your own configuration there.

6. Schema. If db/postgres.sql changed and you create the schema by hand, run the new file as the owner role. Its statements are idempotent.

7. Deploy the new version. A rolling update is possible when the release notes do not say otherwise; all versions that write must agree on the engine's rules, so do not leave mixed versions running longer than the roll-out.

8. Check: /api/health is ok on every replica, with the entry count and head hash you expect; status and verify once more from the operations host; sign in and open a record.

9. Record the version, image digest, the verify output and the checks in your installation record.

Rolling back means deploying the previous version again. Entries written by the new version remain in the trail; if the old version cannot replay them it will report an integrity fault and refuse writes, so test the rollback in step 4 as well when the changelog announces engine changes.