Production readiness · Whole codebase · 2026-09-05

The code is nearly ready. The operations are not.

A different question from rounds 10 and 11, and a different rubric: not whether the code is well written, but whether this can be deployed, operated, diagnosed and recovered. On that measure the library's GA core holds up and the SaaS does not — and almost nothing standing in the way is a code-quality problem.

Library 231k LOC / 106 modules SaaS 22k LOC backend / 187 frontend sources Blockers 6 Method: static analysis, nothing deployed or load-tested
trustweave
6.9/ 10Ready, with scope limits

The eight GA modules are safe to depend on. The other ninety-odd are not, and the project says so plainly in its own maturity matrix — which is the single best readiness artifact in either repository. What blocks a 1.0 is release engineering, not correctness.

Security & access control
8.5
Observability & diagnosability
5.0
Reliability & correctness
7.5
Configuration & data
7.5
Release engineering
5.5
Testing & documentation
7.5
trustweave-saas
5.2/ 10Not ready

Authorization and data integrity are in good shape after two rounds of fixes. But there is no infrastructure definition, no metrics pipeline, no request tracing, and the documented deployment path ships default credentials. You could not diagnose an incident on this service today.

Security & access control
8.0
Observability & diagnosability
3.5
Reliability & scale
4.5
Configuration & data
6.0
Deployment & release
3.0
Testing & documentation
6.0
These numbers are not a regression

Round 11 scored 7.6 and 7.0 on code quality. This assessment asks whether the system can be run in production, which weighs dimensions that a code review barely touches — and where a codebase can be excellent and still unready. Nothing got worse; in fact SA-R11-01 and SA-R11-02 were both closed well since round 11, and I confirmed both. The library moved less because a library's readiness rests on API stability and diagnosability, and neither has changed.

Must fix before production

Blockers

Six. Five belong to the SaaS, and none of them is about the correctness of the code.

Blocker

There is no way to see what the service is doing

PR-01 · SaaS

The observability stack is a single dependency, spring-boot-starter-actuator, and three exposed endpoints. There is no metrics registry, so /actuator/metrics has nothing scraping it. There is no tracing library. There are no correlation identifiers anywhere — a search for MDC, traceId, correlationId and requestId across the whole backend returns nothing, so the 123 log statements cannot be tied to a request, a tenant, or each other. There is no logging configuration file, so output is Spring's default console pattern rather than structured JSON.

There are also no custom health indicators. Health is the framework default, which checks the database and nothing else — not Keycloak, not Accountly, not Kill Bill, not the KMS provider, not the blockchain anchors. Every one of those can fail while the load balancer still sees a healthy instance.

server/build.gradle.kts actuator only — no micrometer-registry-prometheus, no OpenTelemetry application.yml:79-86 exposure: health, info, metrics; no probes group configured grep MDC|traceId|correlationId|requestId → 0 hits grep HealthIndicator → 0 implementations server/src/main/resources/ no logback-spring.xml or log4j2.xml OncePerRequestFilter only DevAuthFilter, which is @Profile("dev")
Fix: add micrometer-registry-prometheus and expose the scrape endpoint on the platform-admin path. Add a OncePerRequestFilter that puts a request id and the resolved organization id into the MDC, and a logback-spring.xml that emits JSON including both. Add health indicators for Keycloak and the billing provider, and enable the liveness/readiness probe groups so orchestration can tell starting from broken.
Blocker

No infrastructure definition, and the documented deploy ships default credentials

PR-02 · SaaS

There is no infrastructure-as-code of any kind — no Terraform, no CDK, no Kubernetes manifests, no CloudFormation. The only deployment artifact is docker-compose.yml, and DEPLOYMENT.md presents it as the recommended path. That file hardcodes POSTGRES_PASSWORD: postgres, KEYCLOAK_ADMIN_PASSWORD: admin, DATABASE_PASSWORD: postgres and an empty KEYCLOAK_CLIENT_SECRET.

This is the same class of defect that SA-R10-06 and SA-R11-02 fixed inside application.yml. Those fixes made the application refuse to start without real credentials; the deployment file then supplies the weak ones from outside, which puts the default back.

Fix: treat docker-compose.yml as a local development file, say so in DEPLOYMENT.md, and source every credential from .env with no fallback. Then write the real target as code. Per the AWS staging design, that means a managed Postgres, a secret manager, and Vault on Fargate for the Ed25519 KMS — none of which exists yet in any committed form.
Blocker

The service is built against SNAPSHOT dependencies

PR-03 · SaaS

Eight dependencies resolve to com.geoknoesis.trustweave:*:1.0.0-SNAPSHOT, including trustweave-all, the KMS providers and the status-list implementations. A SNAPSHOT is mutable by definition: the same build, run twice, can produce different binaries. That removes reproducibility, auditability and the ability to roll back to a known artifact — which for a service whose product is verifiable trust is a particularly awkward place to be. The coordinate also names a version, 1.0.0, that the library has never released; the library is at 0.7.0.

The mitigation in place is genuinely clever and worth keeping: CI pins the library by git revision and verifies a source hash before building. But that guarantees the inputs were reviewed, not that the resolved artifact is immutable or reproducible from a coordinate.

server/build.gradle.kts 8 × com.geoknoesis.trustweave:*:1.0.0-SNAPSHOT trustweave/build.gradle.kts:22 version = "0.7.0" .github/workflows/ci.yml verify-sdk-source.py --revision (pins inputs, not artifacts)
Fix: cut a real library release and depend on the fixed version. This is the same work as TW-R10-06 — once the API surface is validated and frozen, the SaaS gets a version it can pin, and the source-hash check becomes a second line of defence rather than the only one.
Blocker

Nothing survives running more than one instance

PR-04 · SaaS

Two mechanisms assume a single process. The rate limiter is an in-process HashMap behind a single @Synchronized lock, so the effective limit multiplies by instance count — and it is the only control protecting the public verification and claim endpoints. Five of the six @Scheduled jobs have no coordination at all; only the OID4VP verification path takes a database work lease.

The usage-outbox drain is the one to look at first. It selects PENDING and FAILED rows with a plain derived query — no SKIP LOCKED, no pessimistic lock, no claim before the HTTP call — so two instances would post the same usage events concurrently. Each event does carry an idempotency key, so billing correctness rests on Accountly honouring it rather than on anything local. That is a reasonable belt, but there is no braces.

RateLimiter.kt:9-21 private val windows = HashMap<String, Window>(), @Synchronized, per-JVM UsageReporter.kt:30,39 @Scheduled 15s → findByStatusInAndAttemptsLessThan… (no claim) VerificationSession.kt:155 releaseWork() — the one job that is coordinated fly.toml min_machines_running = 1, auto_start_machines = true
Fix: add SKIP LOCKED and a claimed status to the outbox drain — that is the highest-consequence one and a small change. Introduce ShedLock (it only needs the database you already have) for the remaining scheduled jobs, and move the rate limiter behind a shared store before scaling past one machine.
Blocker

The integration suite cannot currently be run

PR-05 · SaaS

The round-11 remediation pass reports that Docker stalled during PostgreSQL container startup and the test worker was killed after five minutes, leaving the integration tests compiled but unverified — and notes that prior full-backend instability has not been cleared. That matters more than usual here, because the strongest recent addition, FlywayMigrationTest, is a Testcontainers test. The check that proves the schema builds from empty is exactly the check that currently cannot run.

To the team's credit this was reported plainly rather than papered over, and the remediation explicitly declined to self-score because of it. But a green pipeline you cannot reproduce locally is not a signal you can deploy on.

Fix: get the Docker environment stable and run the full backend suite end to end before any deploy. Until that happens, treat the migration and boot coverage as unverified regardless of what CI reports.
Blocker

No published artifact can be verified by whoever consumes it

PR-06 · Library

The library publishes to Maven with no GPG signature, no SBOM and no build attestation, and CI runs no SAST. Carried forward unchanged from round 10, but it reads differently under a readiness rubric: this is a security library whose entire proposition is that recipients can verify provenance, distributed by a channel that offers its own recipients none. Signing is required for Maven Central regardless, so it has to happen before any real release anyway.

Fix: GPG signing and cyclonedx-gradle-plugin in the release workflow, plus GitHub artifact attestation. Pairs naturally with PR-03, since both are prerequisites for cutting the version the SaaS needs to pin.
Should fix before scale

Needed, not blocking

Needed

No resilience around any external dependency

PR-07 · SaaS

No retry, no backoff, no circuit breaker anywhere in the backend. Timeouts exist and are sensible where they are set — Keycloak at 5s connect and 10s read, Accountly at 20s, OID4VP admission at 3s — but they are the whole strategy. A transient Keycloak blip fails a user request outright, and a slow billing provider consumes request threads until its timeout.

Fix: Resilience4j on the Keycloak admin client and the Accountly client first — those are on the synchronous request path. The webhook and outbox paths already retry through their own attempt counters.
Needed

fly.toml is stale, non-functional, and carries a password

PR-08 · SaaS

It would not deploy if used. It routes to internal_port = 8080 while the application binds 8081 and both the Dockerfile and its health check use 8081. Its DATABASE_URL is a postgres:// URL, which the JDBC datasource cannot consume. Its Keycloak host is the placeholder keycloak.example.com. And it puts trustweave:trustweave in a committed file as plain environment rather than a Fly secret. Since the deployment direction is AWS, this file is legacy — but it currently reads as a supported target.

fly.toml:10 DATABASE_URL = "postgres://trustweave:trustweave@postgres:5432/trustweave_saas" fly.toml:12 KEYCLOAK_SERVER_URL = "https://keycloak.example.com" fly.toml:17 internal_port = 8080 application.yml:77 port: ${SERVER_PORT:8081} Dockerfile: EXPOSE 8081, healthcheck :8081
Fix: delete it, or fix all four and mark it supported. A broken deployment file is worse than none, because someone will eventually try it.
Needed

Diagnosability of the library depends on exceptions it discards

PR-09 · Library

108 swallowed exceptions with no log line, against only 37 files using a logging framework across 231,000 lines. The verification paths are fail-closed and correct, so this is not a security finding — but under a readiness rubric it is the library's weakest dimension. An integrator whose anchoring silently returns false has no way to learn whether the chain rejected the transaction or the RPC endpoint was unreachable.

Fix: a debug-level log carrying the exception at each swallow point, starting with the anchor plugins where these are network failures presented as clean negatives.
Needed

Retention covers two tables; PII is not encrypted at rest by the application

PR-10 · SaaS

Scheduled purges exist for verification history (90 days, bounded batches, validated range) and expired OID4VCI access tokens. Nothing ages out audit_logs, issued_credentials, failed_webhook_events or blockchain_transactions. Subject records — names, emails, wallet identities — are stored as plain columns; there is no AttributeConverter doing column-level encryption, so confidentiality rests entirely on the database's own at-rest encryption, which no committed infrastructure asserts. No backup or restore procedure is documented anywhere.

Fix: a documented backup and restore drill is the cheapest and most valuable of these. Then retention policies for the remaining tables, and a decision recorded either way on column-level encryption for subject PII.
Needed

No static analysis, dependency updates, or audit on the SaaS

PR-11 · SaaS

Carried forward from round 10 and re-confirmed. No ktlint, detekt or spotless on 22k lines of backend Kotlin holding the tenancy and billing logic; no Dependabot; no npm audit in CI. The library repository has ktlint gated on all 106 modules and Dependabot for two ecosystems, so the tooling gap is between the repositories rather than unfamiliar to the team.

Credit where due

What is already production-grade

Assessed the same way as the gaps, and unusually strong for a codebase at this stage.

The module maturity matrix is the best readiness artifact in either repository. It states outright that publishing a JAR does not imply production suitability, tiers every module across three levels, and lists per-module test counts — labelling two registrar servers “Experimental — untested; verify in your environment” rather than quietly shipping them. Eight modules are GA, twelve Experimental. Very few projects are this honest about their own surface. docs/api-reference/module-maturity.md
Ephemeral key material cannot reach production by accident. KmsProviderConfig classifies in-memory providers as ephemeral and refuses them outside dev, test and local, so an unconfigured deployment fails rather than silently holding issuer private keys in heap. KmsProviderConfig.kt:46-55
Schema ownership is settled and proven. All profiles now run ddl-auto: validate with Flyway authoritative, and FlywayMigrationTest builds the schema from an empty database while deriving its expected table set from @Table annotations, so a new entity without a migration fails on its own. Subject to PR-05, since it needs Docker to run. application.yml:16,24 · FlywayMigrationTest.kt
SA-R11-01 and SA-R11-02 are properly closed, and I verified both. A platform-admin realm role now gates operations and metrics in both filter chains and on the controller. TenantAdministration is a single chokepoint requiring admin plus organization access, through which every user and group lookup is tenant-scoped — so a foreign Keycloak user id resolves to a 404 rather than a grant. That is a structural fix, not a patch. TenantAdministration.kt · RoleService.kt:94-105 · KeycloakSecurityConfig.kt:81,88
The supply-chain control between the repositories — pinning the library by revision and verifying a source hash before the build — is stronger than what most teams do, and the round-11 pass tightened it further with an explicit --revision. PR-03 is about immutable artifacts, not about this check, which should stay. .github/workflows/ci.yml · .trustweave-revision, .trustweave-source-sha256
Idempotency and bounded work are handled thoughtfully where they were considered: a processed-webhook-event table, idempotency keys on usage events, 57 uses of pagination, batch-bounded purges, a database work lease on the OID4VP path, and a container running as a non-root user. ProcessedWebhookEvent · UsageOutbox · VerificationSession.kt:155 · Dockerfile
The reference wallets say what they are. “These examples demonstrate selected flows; they are not production custody products.” Stated in the README rather than left for an integrator to infer. reference-wallet/README.md:5
Scoring

How the numbers were reached

Six dimensions, equally weighted, scored for operability rather than code quality.

Dimension Library SaaS Basis
Security & access control 8.58.0 Both strong after three rounds. Library deducted for artifact provenance; SaaS for the credentials that deployment files put back.
Observability & diagnosability 5.03.5 SaaS: no metrics pipeline, no tracing, no correlation ids, no custom health checks. Library: 108 silent exception swallows and thin logging.
Reliability & scale 7.54.5 Library fails closed consistently. SaaS has single-instance rate limiting, five uncoordinated scheduled jobs, and no retry or circuit breaking.
Configuration & data 7.56.0 SaaS: schema ownership settled and proven; partial retention, no application-level PII encryption, no backup procedure.
Deployment & release 5.53.0 Library: no API-surface control, signing, SBOM or provenance. SaaS: no IaC, SNAPSHOT dependencies, a broken legacy deploy file.
Testing & documentation 7.56.0 Library: 3,975 tests and an exemplary maturity matrix, against 21 untested modules. SaaS: good tests that currently cannot be run.
Overall 6.95.2 Library ready within its stated GA scope; SaaS not ready.
Sequence

The path to ready

Ordered by what unblocks the most, not by severity.

  1. Restore the Docker environment (PR-05). Everything below needs a suite you can actually run, and the migration test is the one that proves the schema.
  2. Instrument the service (PR-01). A Prometheus registry, an MDC filter carrying request and organization ids, JSON logging, and health indicators for Keycloak and billing. This is the difference between operating the service and guessing at it, and it is roughly a day of work.
  3. Write the infrastructure down (PR-02). The AWS staging design already exists on paper; committing it as code also gives you the secret manager that retires the default credentials, and the managed Postgres that makes the backup question answerable.
  4. Cut a real library release (PR-03 with PR-06 and TW-R10-06). Validate and freeze the API surface, sign the artifacts, publish an SBOM, then pin the SaaS to a fixed version. One piece of work closes three findings across both repositories.
  5. Make the outbox drain safe (PR-04). SKIP LOCKED and a claimed status, before anything scales past a single machine. Then ShedLock for the other jobs and a shared rate limiter.
  6. Then the hygiene backlog — ktlint and Dependabot on the SaaS, retries around Keycloak and Accountly, retention for the remaining tables, and the eleven findings carried forward since round 10.