explorer-monorepo/docs/specs/observability/tracing.md

Distributed Tracing Specification

Overview

Distributed tracing for request tracking across services.

Distributed Tracing Strategy

Solution: OpenTelemetry or Jaeger

Implementation:

• Instrument services with tracing
• Propagate trace context
• Collect and store traces
• Visualize in UI

Trace Sampling

Sampling Strategy

Head-Based Sampling:

• Sample rate: 1% of requests
• Always sample errors
• Always sample slow requests (> 1s)

Tail-Based Sampling (optional):

• Sample based on trace characteristics
• More efficient storage

Trace Correlation Across Services

Trace Context Propagation

Headers:

• traceparent (W3C Trace Context)
• tracestate (W3C Trace Context)

Propagation: HTTP headers, message queue metadata

Trace Structure

Trace (request)
  ├── Span (API Gateway)
  │   ├── Span (Explorer API)
  │   │   ├── Span (Database Query)
  │   │   └── Span (Cache Lookup)
  │   └── Span (Search Service)
  └── Span (Response)

Performance Analysis Workflows

Analysis Steps

1. Identify slow requests
2. Trace request path
3. Identify bottlenecks
4. Optimize slow components
5. Verify improvements

References

• Logging: See logging.md
• Metrics: See metrics-monitoring.md