0026. Observability backend strategy

Status: Accepted
Date: 2026-06-22
Supersedes: 0007. Observability strategy

Context

Forge runs GraalVM-native Quarkus services on AWS ECS Fargate at deliberately small task sizes. Operators need production-grade visibility into application behaviour, cross-service request flows, and platform health — not only container liveness.

The platform already instruments services with Micrometer and OpenTelemetry for local development. ADR-0007 did not define an observability backend strategy for metrics, traces, dashboards, and infrastructure monitoring. A decision is required before deploying TEST and PROD.

Application audit logging (business events) remains a separate concern from operational telemetry. Governance evidence (CloudTrail, load-balancer access logs, protected evidence buckets) is defined in ADR-0027.

Decision

Forge will use OpenTelemetry as the instrumentation standard across all environments. Backend targets vary by environment and by signal type — metrics and traces follow separate AWS-recommended pipelines.

Signal backends (deployed TEST and PROD)

Environment posture

Architectural constraints

• No OpenTelemetry Collector sidecar (or shared collector service) on Fargate tasks. Export runs in-process from the application container.
• Forge's observability architecture is designed to operate within the existing Fargate task sizing, avoiding additional resource reservations for collector sidecars.
• Infrastructure metrics and alarms (load balancers, databases, WAF, Lambdas, network flow logs) use CloudWatch and SNS, not AMP. AMP is reserved for application metrics.

Configuration — endpoints, credentials, and profile switches — is environment-specific. Application code does not fork between local and deployed runs.

Consequences

Positive

• Aligns with AWS guidance: AMP for metrics, X-Ray for traces, AMG as the unified operator surface.
• Preserves minimal Fargate footprint and cost profile.
• INT stays lean: no managed observability stack or export charges.
• Clear separation between application telemetry, infrastructure monitoring, and governance evidence — see ADR-0027.
• Local developers retain a full metrics-and-traces loop without AWS dependencies.

Negative

• TEST and PROD operators depend on AMP, AMG, and X-Ray availability and pricing.
• INT cannot validate end-to-end observability pipelines; that validation belongs in TEST.
• In-process export ties telemetry behaviour to application release and resource limits; there is no collector buffer for back-pressure or advanced tail sampling without a future architectural change.

If Forge later requires capabilities such as centralized tail sampling, multi-destination export, or telemetry transformation, this decision should be revisited and an OpenTelemetry Collector re-evaluated.

Alternatives considered

ADOT Collector as a per-task sidecar

Rejected. A collector sidecar would materially increase the minimum resource allocation per task and undermine the platform's native-image and right-sized-task optimisations. Direct in-process export avoids that overhead.

Shared OpenTelemetry Collector service

Rejected. No current requirement for collector-only capabilities (tail sampling, multi-destination fan-out, PII scrubbing in a central hop). Adds operational surface without a stated need.

Application traces to AMP

Rejected. AWS positions AMP for metrics and X-Ray (via OTLP) for traces.

CloudWatch as the application metrics backend

Rejected for application metrics. AMP with PromQL in AMG is the mature, recommended path for Prometheus-compatible application metrics at scale. CloudWatch remains the right choice for infrastructure metrics and alarms.

Full observability stack on INT

Rejected. INT exists to validate integrations, not to host production-like monitoring infrastructure or incur AMP/AMG cost.

References

• ADR-0007: Observability strategy — superseded
• Observability architecture — architecture guide
• Application telemetry — architecture guide