The current evidence does not support an overall score above 9.7. This review identifies concrete implementation defects and the qualification work needed for a defensible 9.8 target. It changes review artifacts only; findings remain open.
Provisional engineering judgment. Six equally weighted categories and equally weighted repositories; joint mean is a planning indicator, not a deployability gate. Historical SDK 9.3 had narrower scope. New SDK reassessment includes the selected provider boundary. No point changes imply statistical precision.
| Aspect | SDK | SaaS | Joint | Assessment |
|---|---|---|---|---|
| Security and access control | 9.0 | 7.5 | 8.25 | SDK: Core controls retained; real custody and host policy remain open. SaaS: KMS profile bypass, fail-open abuse limits and capability logging require correction. |
| Observability and diagnosability | 9.6 | 7.5 | 8.55 | SDK: Prior SDK host telemetry evidence retained; it does not qualify Spring SaaS telemetry. SaaS: Readiness semantics and sensitive path telemetry are incorrect; end-to-end export remains unqualified. |
| Reliability and scale | 8.5 | 6.5 | 7.50 | SDK: Vault response defect and unqualified end-to-end recovery lower the broader assessment. SaaS: Outbox scheduler transaction, lease duration and outage handling remain blockers. |
| Configuration and data | 8.5 | 7.5 | 8.00 | SDK: Strict loaders and ledger audit are valuable, but Vault parsing and host checkpoint ownership are incomplete. SaaS: KMS option wiring, profile composition, recovery and deployment configuration need acceptance evidence. |
| Deployment and release | 8.5 | 6.5 | 7.50 | SDK: Earlier 8.5 retained; no deployed production profile is newly qualified. SaaS: Recipe mismatches and absence of runtime artifact promotion evidence constrain the score. |
| Testing and documentation | 9.0 | 7.0 | 8.00 | SDK: Prior hosted evidence retained; the driver contract defect demonstrates an important provider test gap. SaaS: 298 frontend tests pass, but backend outcome and missing proxy/provider/coverage assurance prevent a high score. |
Arithmetic: SDK 53.1 / 6 = 8.85; SaaS 42.5 / 6 = 7.0833; joint 95.6 / 12 = 7.9667. Display uses one-decimal half-up rounding. Earlier SDK 9.3 is preserved as a narrower historical assessment, not silently rewritten.
Acceptance rule: Zero open P0/P1; both repository means >9.7 unrounded; critical category floors >=9.7; immutable pair and actual declared deployment/custody/recovery gates pass. A high average cannot waive a blocker.
Rubric: 9.8–10 requires qualified declared production behavior and independent operational evidence; 9.0–9.7 has strong engineering with remaining qualification limits; 7.0–8.9 has material implementation/assurance gaps; below 7 has core reliability or release blockers. These are review judgments, not statistical measurements.
Full score recordFailure triage: 442 passed, 111 failed, 3 skipped. Seven failures are stale usage-reporter assertions; 104 are initial context failures or cached cascades from the requestContextFilter bean collision. These are not 111 independent defects. AccountlyDeploymentContractTest (1) and AccountlyLiveContractTest (2) are skipped; real deployment qualification remains open. No fresh backend coverage result is claimed from this failed invocation. Root-cause evidence · Commands and probe reproduction.
SDK HEAD: e0a4464fc6cafa339b94121fd56f33812c6601e4
SaaS HEAD: f31dfeb8af1ec7572d112f110c83990173f3dc9b
SaaS SDK pin: a41d482a96dcdd3962c2a0a95fd1da469f310fa9. Both workspaces contain pre-existing edits; HEAD alone does not identify reviewed content.
The modular SDK and SaaS service separation are useful. The main risks lie where boundaries meet: proxy transactions versus scheduled calls, a local database commit versus a remote billing effect, a driver’s wire representation versus Kotlin types, and a profile name versus actual durable custody.
The reviewed code includes issuer/audience validation, default-deny route rules, tenant ownership checks, atomic token consumption, unique webhook deduplication, failed-webhook recovery storage, bounded local rate-limit state and bulk credential expiry. SDK configuration parsing, ledger audits, hosted test manifests and host telemetry provide substantial prior evidence. None substitutes for the missing integration acceptance tests.
Measure remote calls inside broad transactions, sequential usage batches, scheduler lease expiry, shared-limiter hot rows/key churn and user-sync pagination first. Do not infer production throughput from the earlier one-minute component load test. Recovery must include external billing effects, consumed authorizations and key bindings before admission resumes.
Confidence labels distinguish local reproduction, source-confirmed behavior, design risks and missing qualification. Open each item for impact, correction and acceptance evidence.
The custom @Component RequestContextFilter receives the default bean name requestContextFilter. Spring Boot MVC auto-configuration registers its own bean with that name. The full backend run reproduces BeanDefinitionOverrideException in 11 initial context loads, followed by cached failure-threshold errors in dependent cases.
ApplicationContextSmokeTest and multiple real integration contexts cannot start. This is a code-level bean-name collision, not a missing Docker or identity-provider prerequisite. The 111 failing test cases are not 111 independent defects.
Give the telemetry filter an explicit distinct bean name (or rename the class), preserve Spring’s request-context filter, and keep bean overriding disabled. Re-run real application smoke and all affected integration suites.
Both framework and telemetry filters exist with intended ordering and no duplicate registration. The application boots, MDC is cleaned after errors/async dispatch, and the complete suite passes without enabling bean-definition overriding.
scheduledDrain() invokes drainOnce() on the same bean. Only drainOnce() has @Transactional. The native claim query uses FOR UPDATE SKIP LOCKED. Under Spring’s default proxy transaction mode, this internal call does not open the intended encompassing transaction.
The scheduler path cannot rely on row locks covering delivery and persistence. Whether the provider rejects the query or releases locks early must be established with the real scheduled entry point; neither outcome is acceptable as the claimed concurrency guarantee.
Move work behind a separate proxied worker or explicit transaction boundary. Prefer short claim/finalize transactions with durable leases; do not merely put a long transaction around HTTP delivery.
Invoke the Spring-managed scheduled entry with PostgreSQL, assert the claim transaction, pause two workers at claim/send boundaries, and kill/restart one worker. Verify recovery and provider idempotency.
A five-minute scheduler lease protects up to 100 sequential events with a default 20-second request timeout. One timeout per event alone can exceed 33 minutes. Generic transport and circuit-open failures consume the ten-attempt poison budget; only HTTP 429/503 are exempt.
A long batch can outlive its scheduler lease. When transactions are corrected, remote calls inside the transaction can exhaust DB capacity. A prolonged transient outage can leave valid billable events terminally FAILED and dependent on manual intervention.
Use per-row durable claim tokens/expiry, bounded batches and deadlines, short transactions, remote idempotency, and classified exponential retry with jitter. Expose authenticated redrive and oldest-pending alerts.
Test lease expiry, provider commit followed by lost response, 429/503, repeated connection failures and recovery after more than ten drain ticks. No valid event is silently abandoned or billed twice.
UsageReporterTest stubs findByStatusInAndAttemptsLessThanOrderByCreatedAtAsc, while production calls claimForDelivery. It constructs UsageReporter directly and calls drainOnce(), so it also misses scheduler/proxy semantics.
Existing assertions do not qualify the changed claim path. Merely replacing mock method names would still leave the transaction defect undetected.
Update unit contracts and add real PostgreSQL/Spring proxy integration tests for the scheduled entry point, concurrent claims and retry state transitions.
All existing reporter cases pass against the current method; a deliberately removed transaction or broken claim lease makes an integration test fail.
RateLimiter.check catches every Exception from the shared store, logs a warning, and returns success. This policy applies to public token, claim and redemption paths as well as verification. No dedicated failure-policy metric is emitted by this class.
Counter-table permission loss or a store outage removes abuse limits on sensitive public operations. This is not a JWT/signature bypass, but authentication alone does not replace abuse controls.
Define endpoint-specific failure policy. Fail closed with bounded 503 on sensitive mutations; use an explicitly bounded local fallback only for approved read operations. Add low-cardinality metrics and sampled logs.
Break only the counter-table access while the rest of the service remains available. Sensitive operations reject without downstream mutation; approved read fallback remains bounded across concurrent callers.
The upsert and expiry sweep use Instant.now() from each application instance. Shared rows have no configured maximum, unlike the bounded local map; rejected hits still increment an integer counter.
Clock skew can make nodes disagree about expiry. High-cardinality traffic and a hot key can load the primary DB. No measured failure threshold is claimed by this review.
Use database time for the shared decision, saturate counts, bound statement latency and retention, and define a key-cardinality/pool budget. Consider a dedicated limiter store only if measured DB costs justify it.
Run skewed-clock multi-node tests plus hot-key and high-cardinality load; record p95/p99 latency, pool occupancy, retained rows, cleanup time and rejection correctness.
validateForProfiles allows the in-memory provider whenever any profile is dev/test/local. A local Java probe rejects prod alone but accepts prod+local and staging+local. TrustWeaveConfig is disabled only for test, so local does not disable facade construction.
An accidental profile mix can deploy issuer keys that disappear on restart, leaving persisted issuer identities unable to sign.
Make deployed profiles dominate development allowances, reject incompatible profile combinations, and validate the effective persistent provider before startup.
Table-test empty, unknown, mixed, case-varied and production profiles; boot prod+local and staging+local and require rejection. A permitted persistent profile must retain the same signing identity across restart.
KmsProviderConfig exposes provider-specific options, but TrustWeaveConfig forwards only provider and algorithm to the keys builder. No kmsConfig.options consumer was found.
An operator can supply a documented option without changing the effective provider configuration. A provider may separately read environment settings; this finding does not claim all environment-based setups fail.
Forward typed, validated provider options through the actual factory or reject unsupported configuration explicitly. Publish a secret-safe effective-configuration fingerprint.
A non-default endpoint/namespace is observed by a fake provider; unknown or unused options fail startup and no secret appears in diagnostics.
RequestContextFilter puts raw request.requestURI into MDC HTTP_PATH. Production JSON logging includes httpPath. PublicClaimController serves /api/public/claim/{offerCode}, where offerCode is used to locate the claim offer.
Logs emitted while handling a claim can retain its capability value. requestURI excludes query strings; this finding concerns path tokens and does not assert that every request emits a log.
Use route templates or an allowlisted path redactor before logging. Apply the same rule to access logs, trace attributes and error diagnostics.
Send unique canary offer codes through success and failure cases; captured logs/traces contain no canary while request correlation and route-level metrics remain useful.
The dependency probe treats every status below 500 as UP, including 401, 404 and 429. Keycloak uses a public OIDC discovery URL and participates in readiness.
A deleted realm or incorrect probe URL can remain green after startup. Reachability is useful information, but is insufficient for a functional readiness decision.
Separate reachability from readiness. Require successful, valid discovery metadata for the configured issuer, with bounded timeouts and controlled probe caching; keep liveness independent.
Exercise 401/404/429/500, malformed 200, wrong issuer and valid discovery responses. Readiness changes correctly without a liveness restart storm.
fly.toml routes to 8080 while application.yml defaults to 8081 and the Docker image exposes/checks 8081. The Fly environment supplies no SERVER_PORT override, uses a postgres:// example URL where the datasource expects a JDBC URL, and retains a placeholder Keycloak host.
The committed recipe is not a reproducible production deployment. External secrets or platform overrides may make an existing deployment work; none were inspected here.
Choose one supported port/configuration contract, remove example production fallbacks, require validated datasource/issuer settings, and qualify the actual image through the documented deployment recipe.
Build and boot the image with the declared profile, reach readiness through the configured proxy port, reject missing secrets, and migrate a real supported PostgreSQL instance.
The SaaS pin expects a41d482a96dcdd3962c2a0a95fd1da469f310fa9; the sibling SDK HEAD is e0a4464fc6cafa339b94121fd56f33812c6601e4 with local changes. verify-sdk-source.py rejects this pair. The previously qualified SDK candidate is 5e8dc04c0e45dd51b323b6b3c475d302615715af.
Current local integration tests cannot certify the pinned release pair. This does not prove that SaaS CI against its intended pinned revision fails.
Create immutable reviewed candidates in both repositories, update the pin and reviewed source digest manifest deliberately, and run all integration and artifact gates on that exact pair.
Source verifier, clean builds and hosted integration pass on the same recorded pair; all delivered artifacts and reports carry both SHAs and source digests.
The inspected CI tests backend sources and builds frontend assets, but has no bootJar/container release build, SBOM/provenance attestation, image promotion/rollback qualification or deployed digest verification. Actions use version tags and the Docker base is a floating tag.
Green source tests do not establish the identity or readiness of the image that is deployed. No specific vulnerable dependency is alleged; no fresh vulnerability scan was run.
Build the runtime image once from the qualified pair, pin build inputs, produce SBOM/provenance, enforce a vulnerability policy, verify signatures/digests at promotion, and exercise rollback.
Tampered provenance or mismatched SDK/image digest blocks promotion; an immutable image passes startup, smoke, migration and rollback tests with archived evidence.
SaaS backend coverage verification has a 0.40 floor. Frontend coverage has reporters but no thresholds and CI runs ordinary tests. MSW is configured to warn on unexpected requests; the current passing suite emits network warnings. Previous SDK global coverage was 57.41% line / 40.04% branch.
Passing counts and category scores are not coverage percentages. Proxy, provider response and deployment defects can survive mock-heavy tests. Network warnings need classification rather than blanket suppression.
Define critical-path branch and mutation targets, exercise real transaction/provider contracts, and make unexpected browser requests fail deterministically after fixture cleanup. Retain an explicit skip/discovery manifest.
Critical negative cases fail when their guards are removed; risk-based coverage thresholds are enforced in CI and no unapproved test skips or unexpected requests remain.
syncAllUsers is transactional, fetches the user list, then calls a method that fetches each user remotely and saves it. Per-user exceptions are caught inside the outer transaction. The list call has no explicit pagination at this layer.
Large or slow identity-provider responses can lengthen transactions; a database failure may poison the outer transaction despite per-user error counting. Pagination completeness needs an explicit contract; no undocumented provider page-size assumption is made.
Fetch bounded pages outside DB transactions, persist each bounded unit through a real transaction boundary, checkpoint progress and make retries idempotent.
Sync more than one provider page, inject a mid-page DB failure and slow identity responses, restart and resume, and demonstrate bounded DB occupancy and correct completion counts.
VaultKeyManagementService generateKey/getPublicKey reads keyInfo.data["keys"] and casts it to Map. The pinned driver LogicalResponse.getData() returns Map<String,String>. javap and a local nested-JSON fixture show keys is a java.lang.String; the cast cannot succeed. The compiler reports this at both paths. getDataObject() preserves structured JSON.
A valid nested Vault key response cannot yield a public key through these extraction paths. Key generation can create a provider-side key and then return failure. This is especially material to SaaS staging, which selects Vault.
Parse the driver’s structured response with explicit schema/version/type validation. Cover missing/invalid fields and avoid orphaning/recreating keys on retry. Then qualify the declared Vault algorithm and lifecycle end to end.
A driver-faithful fixture passes generate/get-public-key behavior; malformed versions fail safely. Against an isolated Vault instance, create/sign/independently verify/restart/retrieve/rotate succeeds with stable identity and no private-key leakage.
The SDK custody runbook explicitly says not qualified. Its assessed capability catalog has no supported entries; deployment gating is opt-in outside selected factories and compatibility defaults remain LEGACY. The reference wallet adapters remain experimental.
A strong core SDK score cannot certify all providers, physical authenticators or the SaaS-managed signing path. Broad production-ready claims exceed available evidence.
Declare the exact initial production provider/algorithm/wallet surface, enforce its policy at host construction, and complete real signing, denied access, outage, restart, rotation and authorized recovery evidence. Keep excluded adapters explicitly experimental.
The chosen profile has immutable provider/key identity, independent signature verification, negative authorization and replay tests, recovery approval/audit evidence, and no fallback to ephemeral custody.
Previous SDK tests qualify component WAL recovery and a read-only ledger integrity helper. The helper depends on a separately trusted checkpoint; it does not itself store that checkpoint, authenticate an external payment journal or fence new admission during reconciliation. SaaS recovery of billing, claims and custody together is not evidenced by those tests.
A consistent but stale restore can still be unsafe to resume if external effects and authorization consumption are not reconciled. RPO/RTO and replica fencing remain deployment-specific.
Own checkpoint custody and journal authentication at the host, keep admission fenced until verification/reconciliation completes, and exercise application-wide restore with custody and external effects.
Restore an isolated production-shaped dataset, reject stale/tampered checkpoints and missing WAL, reconcile acknowledged external effects, prove no reused authorization or duplicate billing, and measure agreed RPO/RTO before opening admission.
All 46 tasks are open. S = usually less than one engineer-day; M = roughly 1–3 days; L = multi-day/integration work. These are planning estimates, not delivery commitments. Resource access and soak duration can dominate elapsed time. A task closes only when its acceptance evidence passes on the candidate pair.
Done when: Record supported provider, algorithms, wallet flows, deployment platform, PostgreSQL version, tenant scale and excluded experimental operations.
Done when: Both clean candidate SHAs, reviewed source manifest and source verifier agree; preserve existing local work.
Done when: Faithful nested driver fixtures cover create/get, missing keys, version selection and malformed types.
Done when: Isolated provider create/sign/independent verification/restart/rotation passes; no orphaned retry keys.
Done when: Truth table and actual Spring startup reject prod+local, staging+local and unintended ephemeral configurations.
Done when: Non-default endpoint/namespace is used; unknown/unconsumed options fail startup; diagnostics redact secrets.
Done when: Spring-managed scheduler/proxy tests demonstrate short atomic claim/finalize transactions.
Done when: Two-worker, expired-lease and worker-crash tests preserve claim ownership with fencing tokens and bounded batch runtime.
Done when: Connection outage beyond ten ticks recovers; permanent rejection enters an audited terminal queue; retry delay and Retry-After are bounded.
Done when: Provider commit/response-loss and operator redrive produce exactly one remote charge/effect per event key.
Done when: Counter-only outage causes sensitive mutation rejection and bounded approved read fallback; no blanket allow path.
Done when: Skewed caller clocks cannot reset budgets; counters saturate and TTL/pool limits survive hot-key and churn load.
Done when: Canary path tokens are absent from logs, spans, access logs and error cases while correlation remains intact.
Done when: 401/404/429/500, malformed discovery and issuer mismatch are not UP; liveness remains independent.
Done when: One port/JDBC/issuer contract builds and boots through its actual proxy; missing secrets fail safely.
Done when: Current unit cases pass and a regression removing scheduler transaction/lease correctness is detected.
Done when: Explicitly named telemetry filter coexists with Spring MVC, overriding stays disabled, smoke startup and affected integration cases pass.
Done when: Multi-page provider fixtures plus a failed page and restart demonstrate complete, idempotent recovery.
Done when: Slow identity HTTP does not hold a DB transaction; one failed persistence unit does not roll back unrelated successful units.
Done when: The declared profile rejects unknown/stub/disallowed experimental providers before opening resources; no LEGACY bypass in production entry points.
Done when: Wrong tenant/key/algorithm, expired and replayed proofs fail before provider signing; outage does not select a fallback key.
Done when: Restart/access recovery preserve identity; replacement has independent authorization, revoked old binding and durable audit. Physical-device tests apply only if included in T01.
Done when: Checkpoint authenticity and freshness survive DB compromise/restore; application DB alone cannot rewrite the trust anchor.
Done when: No new authorization is admitted until ledger/checkpoint/external journal verification completes; failure remains fenced.
Done when: Authenticated journal and local outbox reconcile commit/ack-loss and stale restore without duplicate effects or reused authorization.
Done when: Record agreed RPO/RTO, dataset size and timings; missing WAL/corrupt backup/stale checkpoint fail closed; recovered service resumes safely.
Done when: Upgrade from two declared supported schema versions under writes; old/new application compatibility and interrupted migration recovery are explicit.
Done when: Every declared public/admin/tenant mutation has wrong-tenant, wrong-role, expired, replay and identifier-substitution outcomes tied to source and tests.
Done when: Declared DID, webhook and provider clients reject disallowed destinations/redirects and bound size/time; tests use actual host integration.
Done when: A real incoming request produces correlated outgoing provider spans with trace propagation and authenticated export; prove manual RestClient construction receives instrumentation.
Done when: Oldest outbox age, leased/failed rows, limiter degradation, recovery fence and reconciliation failures have bounded labels and verified alerts.
Done when: Availability, latency, signing and delivery objectives have error budgets, accountable owners and tested notification acknowledgement.
Done when: Access controls and retention are enforced in the selected backend; sensitive canaries never reach storage; cardinality/load limits are measured.
Done when: Agree workload first; run proposed 72-hour steady/peak soak and burst/fault phases, tracking p99, saturation, queue age, DB growth and recovery. This is a target, not a completed measurement.
Done when: Document tested tenant/data/request limits, capacity headroom and bounded degradation, including KMS/IdP/billing outages.
Done when: Critical auth/custody/claim/recovery branches have explicit agreed floors (proposed 90% line, 85% branch) and meaningful mutation checks; exceptions are reviewed, not hidden.
Done when: Classify current network warnings, add missing fixtures and fail unexpected requests without the prior worker serialization problem; CI runs coverage with agreed floors.
Done when: Test IDs/counts, failures, documented optional skips and required provider profiles are archived and checked against the immutable candidate pair.
Done when: bootJar, frontend assets and OCI image are built from the candidate pair; production-profile startup and smoke tests run against that image.
Done when: Actions/base image are immutable; dependency/image scans enforce a defined policy; SBOM and provenance identify both repository commits.
Done when: A modified image, wrong source pair or untrusted attestation is rejected; the tested digest is the promoted digest.
Done when: Promote a qualified digest with explicit health/error-budget gates and demonstrate rollback with the supported schema compatibility window.
Done when: Setup, configuration precedence, limits, provider scope, backup, rotation, incident and rollback instructions match tested commands and evidence.
Done when: An operator other than the author executes restore, custody outage, stuck-delivery redrive and rollback using only published runbooks; gaps are fixed.
Done when: All P1 findings are closed by behavior tests, scope exclusions are explicit, and evidence is tied to the exact released candidate pair.
Done when: Each repository is above 9.7 unrounded, every critical category is at least 9.7, and production acceptance gates pass. Target 9.8; no points are awarded simply for adding documentation/tests.
This was a targeted source and verification review of both repositories, not an exhaustive penetration test. No actual production deployment, live managed custody, physical authenticator, fresh vulnerability scan, production-scale soak or complete recovery drill was executed. The sibling policy engine is outside this request. Existing local code changes were preserved; this review adds reports and local diagnostic evidence.
Spring’s default transaction proxies intercept calls that enter through the proxy; same-bean invocation does not activate the annotated method’s transaction. Spring transaction documentation.
ShedLock’s maximum lock duration is a lease limit; a task exceeding it can execute concurrently. ShedLock documentation.
The score should be reassessed after behavior changes and production-profile evidence, not increased because this report is more detailed.