Library telemetry
Until 11 September 2026 the SDK’s instrumentation stopped at the HTTP boundary. A host running the
embedded servers got request spans, protected metrics and an X-Request-ID, but the work
underneath — resolving a DID, signing with a KMS, reading a wallet, verifying a credential —
emitted nothing. A slow request could not be attributed, and a failed verification could not be
joined to a cause. This is the missing half.
Shape
The SPI lives in :common and depends on nothing but the Kotlin standard library, so the core
modules do not acquire OpenTelemetry because a host wants tracing. :observability adapts it.
1
2
3
4
did-core · kms-core · wallet-core · credential-api
│ org.trustweave.core.telemetry.Telemetry (no-op until installed)
▼
LibraryTelemetry (:observability) ──▶ OpenTelemetry spans + Prometheus
Turning it on
1
2
3
4
5
6
val library = LibraryTelemetry(openTelemetry)
Telemetry.install(library)
val server = StatusListServer()
.withObservability(HostObservability(telemetry, metricsToken, library = library))
.withAuthentication(HostAuthentication.bearerToken(adminToken))
Passing library to HostObservability does two things: the library scrape is appended to
/internal/metrics, and each request’s id is placed in the coroutine context so library events
carry it. Nothing is recorded until Telemetry.install is called; the cost before that is one
volatile read per operation.
KMS is instrumented by wrapping, because there is no single implementation to instrument:
1
val kms = AwsKeyManagementService(...).withTelemetry()
What is emitted
trustweave_library_operations_total{operation,outcome} — counter
trustweave_library_operation_seconds{operation} — histogram, buckets 5ms to 30s
Both labels are closed enums, so the series count is fixed no matter what traffic arrives.
operation |
Emitted from |
|---|---|
did_resolve |
CachingDidResolver.resolve |
did_create, did_update, did_deactivate |
TelemetryDidMethod, applied by DidMethodRegistry.register |
kms_generate_key, kms_sign |
TelemetryKeyManagementService |
wallet_store, wallet_get, wallet_query, wallet_delete |
TelemetryWallet, applied when the wallet DSL builds a wallet |
credential_issue, credential_verify, credential_verify_presentation |
DefaultCredentialService |
credential_revocation_check |
DefaultCredentialService.status |
Every declared operation appears in this table. scripts/check-telemetry-operations.py fails
the build if a value is declared with no main-source emitter, or emitted without being declared, so
the enum stays a contract about what the library reports rather than a list of intentions. A value
nothing emits is worse than no value: a host charting it sees a permanent zero series and cannot
tell “not instrumented” from “never happened”. Removing the value is always an acceptable fix.
There is no kms_verify: KeyManagementService has no verify operation to report.
did.method is attached to the DID operations, kms.algorithm to the KMS ones, and
wallet.read (list or query) distinguishes the two shapes of wallet_query. Nothing that
identifies a subject is ever attached — not a DID, not a credential id, not a result count.
Outcomes
outcome |
Meaning |
|---|---|
success |
Completed and returned a successful result |
rejected |
Completed correctly, and the answer was no — an invalid credential, an unresolvable DID |
failure |
Threw |
cancelled |
The caller’s coroutine was cancelled |
rejected is deliberately not failure. A credential that fails verification is the library
working; a host that pages on it will page on ordinary traffic. Alert on failure, chart
rejected as a rate and alert only on a change in that rate.
cancelled is also separate: a cancelled caller is not a failed operation, and conflating the two
is what makes cancellation bugs invisible.
Reason codes
reason is the exception’s simple class name, never its message. Messages carry DIDs, file paths,
key identifiers and provider error text, and reason codes end up as span attributes. If you need
the message, log it on your side from the exception you already caught.
Span attributes
trustweave.operation, trustweave.outcome, trustweave.reason, trustweave.request_id, plus
operation attributes: did.method, kms.algorithm. All are low-cardinality by construction.
A library span is emitted at the end of the operation, back-dated by the measured duration — the library already timed it, and re-timing at the bridge would double-count.
What this does not do
- It does not install a logging backend, an exporter or a global OpenTelemetry instance. The host owns all three.
- It does not record arguments, subjects, key material or credential contents.
- It does not replace the host request surface in host/README.md; it sits under it.
- Wallet operations are declared in the enum but not yet wired to call sites.