Live custody and key-loss qualification
Status: software PKCS#11 boundary qualified in CI; executable AWS KMS live
qualification is implemented and awaits an identified test-environment run. CI provisions an isolated SoftHSM2 token and verifies P-256 key
generation, lookup, signing, deletion, unsupported-algorithm handling and reuse of
an existing key after recreating the KMS client. These named tests are mandatory in
config/testing-contract.json; the skip policy does not permit them to be skipped.
SoftHSM2 does not qualify a hosted or physical HSM, provider IAM, production audit
delivery or dual-control recovery. The protected AWS KMS custody qualification
workflow now exercises real signing, independent verification, denied access,
client restart, and replacement-key continuity with immutable resource IDs. It does
not become qualification evidence until it passes against recorded non-production
resources. Accountly billing validation does not qualify custody. The reference
wallet’s managed and passkey adapters remain experimental; see the implemented
contract.
Identify the test boundary
Before execution, record these non-secret values in the evidence record:
- Provider, account/project/subscription, region and immutable key resource ID.
- Existing credential profile or workload identity; never include credential values.
- Key algorithm, public-key fingerprint and service endpoint under test.
- Disposable tenant, wallet and key; explicit scope for reversible access revocation.
- Application commit, deployed artifact digest, configuration revision and test time.
- Recovery operator identity, independent approver and recovery policy version.
Use an isolated non-production resource. Do not infer permission to disable a shared key, change an account-wide policy or schedule deletion. A provider alias alone is insufficient: record the resolved immutable key ID before and after each exercise. The signing principal and recovery administrator must be distinct.
Configure the AWS KMS qualification environment
Create the protected GitHub environment aws-kms-qualification, require an
independent reviewer, and set these environment variables (not secrets):
| Variable | Value |
|---|---|
AWS_KMS_QUALIFICATION_ROLE_ARN |
IAM role trusted only by this repository’s GitHub OIDC subject and protected environment |
AWS_KMS_QUALIFICATION_REGION |
Region containing all three test keys |
AWS_KMS_PRIMARY_KEY_ARN |
Immutable ARN of a pre-provisioned P-256 SIGN_VERIFY key |
AWS_KMS_REPLACEMENT_KEY_ARN |
Immutable ARN of a distinct P-256 replacement key |
AWS_KMS_DENIED_KEY_ARN |
Immutable ARN on which the role has an explicit deny for signing and public-key access |
The role needs only kms:DescribeKey, kms:GetPublicKey, and kms:Sign on the
primary and replacement keys. Deny those operations on the denied key. Do not grant
kms:CreateKey, kms:DisableKey, kms:PutKeyPolicy, kms:ScheduleKeyDeletion, or
alias mutation. GitHub exchanges its OIDC token for short-lived credentials; no AWS
access key is stored in GitHub.
Run .github/workflows/aws-kms-qualification.yml manually after reviewing the
candidate commit and resource scope. The workflow fails when any identifier is
absent, an alias is supplied, the three resources are not distinct, a check is
skipped, or evidence does not match the candidate commit. The retained JSON contains
key ARNs and public-key fingerprints but no challenge, signature, credential, or
provider token.
The live test is read-only. It simulates lost-key replacement by proving a new binding with the replacement key and proving that the original public key remains available for historical verification after a fresh SDK client starts. It does not disable or destroy the old key and therefore does not claim a destructive recovery exercise.
Acceptance matrix
| Gate | Exercise | Required evidence |
|---|---|---|
| Provider identity | Resolve the identified key through the configured SDK and retrieve its public key | Exact resource ID, algorithm, public-key fingerprint and provider request ID |
| Real signing | Sign a fresh random challenge through TrustWeave; verify independently with the public key; reject altered message/signature | Challenge digest, verification results and correlated provider audit event; no private material |
| Authorization | Attempt signing as the wrong tenant, wrong holder and an unprivileged principal | All rejected before signing; audit trail proves no unauthorized provider sign operation |
| Protocol | Test audience, expiry, nonce replay, key ID and algorithm substitution through the deployed endpoint | Exact expected outcomes, durable challenge consumption and independently verified JWS |
| Provider outage | Deny access to the disposable key or isolate the test service, then request signing | Bounded failure, no alternate-key/local-key fallback and no success response |
| Access recovery | Restore authorized access using the recovery administrator, restart the service and retry with a fresh challenge | Recovery audit event, unchanged key fingerprint and successful fresh proof; old nonce still rejected |
| Lost-key replacement | Make the old disposable key unavailable; enroll a distinct replacement through the recovery workflow | Independent recovery authorization, new fingerprint, old binding revoked, credential reissuance where binding changes |
| Durable recovery | Restore application state into an isolated instance and replay pre-recovery challenges | Old authorizations rejected; recovery completion and key binding survive restart; no duplicated signing authorization |
| Incident visibility | Trigger the test failure/recovery alerts | Correlated application/provider events, alert delivery and acknowledgement timestamps |
Revoking access and restoring it demonstrates access recovery, not recovery of a destroyed private key. Replacement-key recovery proves continuity through a new identity binding; it cannot recreate signatures from a lost non-exportable key. Do not silently relabel old did:key credentials or accept both old and new holder bindings without an explicit, tested policy.
Execution and evidence rules
- Capture the identified resource and approved mutation scope. Confirm the
application endpoint implements the server obligations in
CUSTODY.md. - Run baseline signing and all rejection cases before mutating access.
- Record the recovery start and deny signing only for the disposable resource. Preserve the original policy/configuration for restoration.
- Exercise recovery, independently verify the new proof and test all old authorizations again. Compare immutable IDs and public-key fingerprints.
- Restore test access/configuration in a cleanup step even if assertions fail. Record cleanup failure as a failed exercise requiring operator attention.
- Retain sanitized results plus immutable references to restricted audit records. Do not publish bearer tokens, wallet backups, private keys or full credentials.
Each gate records passed, failed, or not_run, its command/driver, start/end
times, expected/actual outcome and evidence reference. Missing inputs, unavailable
services and skipped tests are not_run, never a pass. Report recovery duration
as an observation for this resource; do not turn it into a production RTO promise.
Remaining application work
The current reference-wallet adapters do not implement a production managed signing endpoint, durable enrollment/recovery service or custody selection in the existing Ed25519 UI. Those components must be implemented and exercised before end-to-end custody can be qualified. A direct provider SDK test can close only the provider identity/signing gate. Physical passkey and sync-account recovery require their own device/provider evidence and are not covered by a cloud KMS exercise.
The Accountly repository contains a protected Staging production qualification
workflow and scripts/qualify-deployment.py. It exercises issuance and verification
on both sides of a did:web key rotation, cross-tenant rejection, mixed authenticated
load, process restart, isolated backup restore, and Alertmanager-sourced on-call
acknowledgement. This is an executable qualification contract. It becomes evidence
only after a successful run records the identified deployment revision, image digest,
KMS resource, database version, region, recovery operations and alert acknowledgement.
Accountly is TrustWeave SaaS billing/subscription infrastructure; it is not a wallet,
signing service, or custody provider, so its run cannot close the SDK custody gate.