Library telemetry

Until 11 September 2026 the SDK’s instrumentation stopped at the HTTP boundary. A host running the embedded servers got request spans, protected metrics and an X-Request-ID, but the work underneath — resolving a DID, signing with a KMS, reading a wallet, verifying a credential — emitted nothing. A slow request could not be attributed, and a failed verification could not be joined to a cause. This is the missing half.

Shape

The SPI lives in :common and depends on nothing but the Kotlin standard library, so the core modules do not acquire OpenTelemetry because a host wants tracing. :observability adapts it.

1
2
3
4
did-core · kms-core · wallet-core · credential-api
        │  org.trustweave.core.telemetry.Telemetry   (no-op until installed)
        ▼
   LibraryTelemetry  (:observability)  ──▶  OpenTelemetry spans + Prometheus

Turning it on

1
2
3
4
5
6
val library = LibraryTelemetry(openTelemetry)
Telemetry.install(library)

val server = StatusListServer()
    .withObservability(HostObservability(telemetry, metricsToken, library = library))
    .withAuthentication(HostAuthentication.bearerToken(adminToken))

Passing library to HostObservability does two things: the library scrape is appended to /internal/metrics, and each request’s id is placed in the coroutine context so library events carry it. Nothing is recorded until Telemetry.install is called; the cost before that is one volatile read per operation.

KMS is instrumented by wrapping, because there is no single implementation to instrument:

1
val kms = AwsKeyManagementService(...).withTelemetry()

What is emitted

trustweave_library_operations_total{operation,outcome} — counter trustweave_library_operation_seconds{operation} — histogram, buckets 5ms to 30s

Both labels are closed enums, so the series count is fixed no matter what traffic arrives.

operation Emitted from
did_resolve CachingDidResolver.resolve
did_create, did_update, did_deactivate TelemetryDidMethod, applied by DidMethodRegistry.register
kms_generate_key, kms_sign TelemetryKeyManagementService
wallet_store, wallet_get, wallet_query, wallet_delete TelemetryWallet, applied when the wallet DSL builds a wallet
credential_issue, credential_verify, credential_verify_presentation DefaultCredentialService
credential_revocation_check DefaultCredentialService.status

Every declared operation appears in this table. scripts/check-telemetry-operations.py fails the build if a value is declared with no main-source emitter, or emitted without being declared, so the enum stays a contract about what the library reports rather than a list of intentions. A value nothing emits is worse than no value: a host charting it sees a permanent zero series and cannot tell “not instrumented” from “never happened”. Removing the value is always an acceptable fix.

There is no kms_verify: KeyManagementService has no verify operation to report.

did.method is attached to the DID operations, kms.algorithm to the KMS ones, and wallet.read (list or query) distinguishes the two shapes of wallet_query. Nothing that identifies a subject is ever attached — not a DID, not a credential id, not a result count.

Outcomes

outcome Meaning
success Completed and returned a successful result
rejected Completed correctly, and the answer was no — an invalid credential, an unresolvable DID
failure Threw
cancelled The caller’s coroutine was cancelled

rejected is deliberately not failure. A credential that fails verification is the library working; a host that pages on it will page on ordinary traffic. Alert on failure, chart rejected as a rate and alert only on a change in that rate.

cancelled is also separate: a cancelled caller is not a failed operation, and conflating the two is what makes cancellation bugs invisible.

Reason codes

reason is the exception’s simple class name, never its message. Messages carry DIDs, file paths, key identifiers and provider error text, and reason codes end up as span attributes. If you need the message, log it on your side from the exception you already caught.

Span attributes

trustweave.operation, trustweave.outcome, trustweave.reason, trustweave.request_id, plus operation attributes: did.method, kms.algorithm. All are low-cardinality by construction.

A library span is emitted at the end of the operation, back-dated by the measured duration — the library already timed it, and re-timing at the bridge would double-count.

What this does not do

  • It does not install a logging backend, an exporter or a global OpenTelemetry instance. The host owns all three.
  • It does not record arguments, subjects, key material or credential contents.
  • It does not replace the host request surface in host/README.md; it sits under it.
  • Wallet operations are declared in the enum but not yet wired to call sites.

This site uses Just the Docs, a documentation theme for Jekyll.