Performance qualification protocol
| Document id | CS-VAL-08 |
| Title | Performance qualification (PQ) protocol |
| Version | 1.0 |
| Applies to | Countersign 1.0.0 |
| Status | Template supplied with the software; to be completed, reviewed and approved by the regulated user |
Approval of the completed document:
| Role | Name | Signature | Date |
|---|---|---|---|
| Prepared by | |||
| Reviewed by | |||
| Approved by (quality assurance) |
1. Purpose
The PQ documents that the system, with your approved configuration, your identity provider, your roles and your procedures, supports your business process end to end and meets your user requirements under conditions of routine use. The OQ showed that the software's rules work; the PQ shows that your configuration of them is the right one.
This is a template. The scenarios in section 5 are examples to adapt; the scenarios you execute must come from your own process and be approved before execution.
| Item | Entry |
|---|---|
| Environment (test with production configuration, or production before release) | |
| Application version and commit | |
Configuration id, revision and SHA-256 (from countersign status) |
|
| Role map version | |
| SOP versions used | |
| Start and end date |
2. Prerequisites
| No. | Prerequisite | P/F | Initials, date |
|---|---|---|---|
| 2.1 | IQ and OQ of the environment are approved; deviations closed or assessed as not affecting the PQ | ||
| 2.2 | The configuration under test is the approved one: the SHA-256 printed by countersign status equals the one in its approval |
||
| 2.3 | Key users executing the PQ are trained on the User guide and the draft SOPs | ||
| 2.4 | Key users have personal accounts with the roles they will hold in production | ||
| 2.5 | Test data representative of production is defined (section 4) | ||
| 2.6 | This protocol, with your scenarios, is approved |
3. Configuration review
Before running scenarios, review the configuration against the user requirements. Attach the output of
countersign config show.
| No. | Check | Expected | Actual | P/F | Initials, date |
|---|---|---|---|---|---|
| 3.1 | Every record type in the user requirements exists, with the required fields, types and rules (required, unique, range, pattern, choices) | As specified | |||
| 3.2 | The hierarchy (parent and group references) matches the business structure; the field order gives the intended inheritance order (a group reference placed before the parent reference is consulted first) | As specified | |||
| 3.3 | Role and permission matrix: for each role, the author and approve lists equal the approved matrix |
As specified | |||
| 3.4 | No role has * in author or approve unless the matrix says so |
As specified | |||
| 3.5 | import and configure are granted only to the role named in the plan |
As specified | |||
| 3.6 | For each record type at least two people can write or approve so that four eyes can always be satisfied, including during absences | At least two authors or one author and one separate approver per type | |||
| 3.7 | The role map assigns each provider group to the intended role id | As specified |
Complete the permission matrix as executed evidence (one row per role):
| Role id | May write (record types, option levels) | May approve | Import | Configure | People holding it (count) |
|---|---|---|---|---|---|
4. Test data
| Item | Entry |
|---|---|
| Source of test data (synthetic, anonymised copy, migrated subset) | |
| Volume loaded for the PQ (records per type, audit entries) | |
How it was loaded (manual entry, countersign import with mapping ______) |
|
| Confidentiality constraints |
5. Scenarios
Each scenario follows this form. Copy the block per scenario and number them PQ-01 onwards. The examples show the level of detail expected; replace the bracketed parts.
Scenario form
| Id and title | |
| User requirements covered | |
| SOP exercised | |
| Roles and people | |
| Starting condition | |
| Steps | |
| Expected result | |
| Acceptance criteria | |
| Actual result (record ids, entry numbers, screenshots) | |
| Pass / Fail | |
| Executed by, date | |
| Reviewed by, date |
Example PQ-01: a new record from request to effect
| User requirements covered | UR-01, UR-02, UR-04, UR-06, UR-15 |
| Roles and people | [Author role], [approver role]: two real users |
| Starting condition | [A real business trigger, for example an approved request form no. ___] |
| Steps | 1. The author creates [record type] with the data of the request, with a note referring to the request. 2. The author checks and confirms. 3. The approver finds the pending item without being told where it is, compares it with the request, and approves with the meaning your SOP prescribes. 4. A reader looks the record up |
| Expected result | The record is in effect with exactly the requested data; the history names author and approver; the audit trail has three entries with the note and the meaning |
| Acceptance criteria | The users complete the scenario with the SOP and the user guide alone; data equals the request; elapsed time is acceptable to the process owner |
Example PQ-02: change of an approved record, rejected once
| User requirements covered | UR-02, UR-03, UR-04 |
| Steps | 1. The author changes [field] of an approved record with a reason. 2. During the pending period a reader confirms which value is in effect. 3. The approver finds an error and steps the version back with a reason. 4. The author corrects and confirms again. 5. The approver approves |
| Expected result | The previous value stays in effect until step 5; afterwards the new value is in effect and the old version is kept as superseded; the audit trail tells the whole story, including the rejection and its reason |
| Acceptance criteria | A person who was not involved can reconstruct who did what, when and why from the record's history alone |
Example PQ-03: an option set at several levels
| User requirements covered | UR-08 |
| Steps | 1. For [option] set values at [top level], [group], [single record] through the workflow. 2. For three chosen records, predict the effective value and its source on paper from the business rules. 3. Compare with what the application shows. 4. Retire the value at the single record through the workflow and compare again |
| Expected result | Prediction and display agree for every record, including the source; after step 4 the next level's value applies |
| Acceptance criteria | Full agreement; the inheritance order is the one the business intends |
Example PQ-04: taking a record out of use
| User requirements covered | UR-09 |
| Steps | 1. Try to retire a [parent record] that still has live records under it. 2. Move or retire the dependants through the workflow. 3. Retire the parent, with approval. 4. Look the retired record and its history up |
| Expected result | Step 1 is refused with the list of dependants; after step 3 the record is shown as retired, can no longer be referenced by new records, and its history is complete |
| Acceptance criteria | The SOP's sequence works without workarounds |
Example PQ-05: absence and segregation of duties
| User requirements covered | UR-02, UR-06 |
| Steps | 1. Simulate the absence of the usual approver for [record type]. 2. The author confirms a change. 3. The deputy named in the SOP approves. 4. Try to have the change approved by a person outside the approved matrix |
| Expected result | The deputy can approve; nobody else can |
| Acceptance criteria | Every record type has a working deputy arrangement that does not require sharing accounts or granting roles ad hoc |
Example PQ-06: periodic audit trail review
| User requirements covered | UR-04, UR-05, UR-13, UR-14 |
| SOP exercised | Audit trail review; head-hash anchoring |
| Steps | 1. The reviewer performs the review of the PQ period following the SOP and the checklist in data integrity, section 9. 2. The reviewer exports the trail and files the copy. 3. The system owner records an anchor and verifies the previous one |
| Expected result | The review record is complete; every cli entry and every step back is explained; countersign verify --expect-entry <n>:<hash> passes for the previous anchor |
| Acceptance criteria | The review can be completed in the time planned for it and finds the anomalies seeded for the exercise (for example an approval seconds after confirmation, a step back without a useful reason) |
Example PQ-07: joiner, mover, leaver
| User requirements covered | UR-06, UR-07 |
| SOP exercised | User access and role management |
| Steps | 1. Request and grant a role for a new user through the SOP. 2. The user signs in and performs an action the role allows and one it does not. 3. Change the user's role. 4. Withdraw access. 5. The user tries to sign in |
| Expected result | Permissions follow the identity provider at the next sign-in; after withdrawal sign-in is refused by the provider |
| Acceptance criteria | Time from approved request to effective change is within the SOP's limit; the delay until a running session ends is understood and acceptable |
Example PQ-08: backup, restore and integrity fault drill
| User requirements covered | UR-10, UR-14 |
| SOP exercised | Backup and restore; integrity-fault handling; business continuity |
| Steps | 1. Restore last night's backup into a separate database. 2. Run countersign verify --expect-head <hash anchored at the time of the backup>. 3. Identify, from the audit trail export or the anchor register, which entries were written after the backup. 4. Walk through the integrity-fault SOP as a table-top exercise using the outputs of OQ case C-02 |
| Expected result | The restore verifies; the lost entries can be named; responders know their steps |
| Acceptance criteria | Recovery time and recovery point are within your requirements |
Example PQ-09: configuration change under change control
| User requirements covered | UR-12 |
| SOP exercised | Configuration change control |
| Steps | 1. Raise a change for a realistic configuration change (for example a new choice in a selection field). 2. Validate and review the document; reviewer signs the change record with the SHA-256. 3. Apply in the test environment with countersign config apply. 4. Verify with countersign status and by using the new choice in a record |
| Expected result | The new revision is in effect with the reviewed hash; existing records are unaffected; one configure entry names the operator, the reviewer (inside the command) and the reason |
| Acceptance criteria | The SOP can be followed as written |
Example PQ-10: migration rehearsal (if data is migrated)
| User requirements covered | UR-11 |
| SOP exercised | Migration |
| Steps | 1. Run countersign import --mapping … --dir … --dry-run --report … on a full extract. 2. Review refused rows. 3. Run the import in the test environment. 4. Reconcile totals and compare a sample of [n] records field by field against the source |
| Expected result | Rows read = imported + already present + refused for every record type; every refusal explained; sample identical |
| Acceptance criteria | Zero unexplained differences |
6. Load and volume
6.1 What to expect from the design
State these facts in your assessment; they follow from how the software is built.
- Each server process keeps the complete replayed state (all records, all versions, all audit entries) in memory.
- On start, a server reads the whole audit trail and re-runs every command to rebuild and check the state. The
command-line tool does the same for
status,verify,export,importandconfig. - Re-running a command includes the same validation as the original (uniqueness checks look at the other records of
the type), and each accepted entry copies internal lists. Start-up and
verifytime therefore grow faster than linearly with the number of entries. - The browser fetches the audit trail through
/api/entriesand rebuilds the same state, so the first load of the application also grows with the trail. - Every request asks the store for its head before answering; a write is one insert.
- There is no pagination of the trail on the server, no archiving of old entries and no snapshot to start from.
The design is suited to configuration and master data: volumes in the order of tens of thousands of audit entries. It is not designed for transactional volumes. The project publishes no performance guarantee. If you expect more than a few tens of thousands of entries over the system's life, measure before you decide, with the method below.
6.2 Your expected volume
| Quantity | Now | In 5 years | Basis of the estimate |
|---|---|---|---|
| Records (all types) | |||
| Changes per month | |||
| Audit entries (about three per new record, two per change of an approved record) | |||
| Concurrent users at peak | |||
| Application servers |
6.3 Measurements
Load a test database to the five-year volume with a margin of ______ % (for example with countersign import and
synthetic files, or by replaying an anonymised production export), then measure on hardware equal to production.
| No. | Measurement | How | Your limit | Result | P/F |
|---|---|---|---|---|---|
| 6.3.1 | Server start until /api/health answers 200 |
Restart the service; poll the endpoint | |||
| 6.3.2 | countersign verify duration |
time countersign verify |
|||
| 6.3.3 | Memory of the server process after start and after one hour of use | Operating-system tools | |||
| 6.3.4 | First load of the application in the browser on a typical workstation | Browser developer tools, cold cache | |||
| 6.3.5 | Response time of one approval | Browser developer tools, median of 20 | |||
| 6.3.6 | [n] users writing during the same minute | Scripted or by hand; check that every action ends in exactly one entry, without gaps, and no user sees an error other than a refusal by the rules | |||
| 6.3.7 | Export of the whole trail | time countersign export --what audit --out … |
|||
| 6.3.8 | Backup and restore duration at this volume | Per backup SOP |
If a limit is not met, the options are operational, not configurable: more memory, fewer but larger servers, a longer start-up allowance in the deployment, or a decision that the use does not fit the tool.
7. Acceptance criteria
| No. | Criterion |
|---|---|
| 1 | Every scenario passed, or deviations are closed |
| 2 | Every applicable user requirement, including your own UR-U requirements, is covered by at least one passed scenario or by a referenced OQ case |
| 3 | The configuration SHA-256 at the end of the PQ equals the approved one (no unrecorded change during testing) |
| 4 | Every measurement of section 6.3 meets your limit, or the deviation is accepted |
| 5 | After the PQ, countersign verify reports the chain intact and all entries reproduced |
| 6 | The SOPs exercised could be followed as written, or their corrections are approved |
8. Coverage of user requirements
| User requirement | Scenario(s) or OQ case(s) | Result |
|---|---|---|
| UR-01 | ||
| UR-02 | ||
| UR-03 | ||
| UR-04 | ||
| UR-05 | ||
| UR-06 | ||
| UR-07 | ||
| UR-08 | ||
| UR-09 | ||
| UR-10 | ||
| UR-11 | ||
| UR-12 | ||
| UR-13 | ||
| UR-14 | ||
| UR-15 | ||
| UR-U-… |
9. Deviations
| No. | Scenario | Description | Classification | Resolution | Closed (initials, date) |
|---|---|---|---|---|---|
10. Test data after the PQ
If the PQ ran on the production installation, its entries stay in the audit trail for good: nothing can be deleted. Decide beforehand: run the PQ on a test installation with the production configuration (recommended), or accept and label PQ records in production and retire or remove them through the workflow afterwards. Decision: ______________.
11. Conclusion
| Statement | Yes / No |
|---|---|
| All acceptance criteria are met | |
| The system is fit for its intended use and is recommended for release |
| Role | Name | Signature | Date |
|---|---|---|---|
| Executed by (key users) | |||
| Process owner | |||
| Approved by (QA) |