OpenTelemetry End-to-End — A 2026 Guide for Backend Engineers

OpenTelemetry has won the wire-format wars. By 2026 it’s the default protocol for traces, metrics, and increasingly logs. If you’re building observability into a service — not bolting it on later — this is the playbook.

This post is the practical end-to-end. The collector model, instrumenting services in Python/Go/Rust, sampling, context propagation, and the shape that scales from a single service to a fleet.

The OTel mental model

OpenTelemetry has three pillars and one connector:

Traces — request paths through services. Think distributed printf with timing.
Metrics — counters, gauges, histograms. Aggregate behavior.
Logs — events with timestamps. The raw record.
Context propagation — the trace ID and parent span ID that flow through every layer so all of the above can be correlated.

The connector matters most. The unique value of OTel isn’t traces or metrics individually — those existed before. It’s that they all share the same trace context, so your Grafana dashboard, your trace explorer, and your log search tell the same story about the same request.

Architecture

┌──────────┐   ┌──────────┐   ┌──────────┐
│  Python  │   │   Go     │   │  Rust    │
│ service  │   │ service  │   │ service  │
└─────┬────┘   └─────┬────┘   └─────┬────┘
      └──────────────┴───────────────┘
                     │ OTLP (gRPC or HTTP)
                     ▼
             ┌────────────────┐
             │ OTel Collector │  ← receivers / processors / exporters
             └───────┬────────┘
                     │
       ┌─────────────┼─────────────────┐
       ▼             ▼                 ▼
   ┌───────┐  ┌────────────┐    ┌──────────┐
   │ Tempo │  │ Prometheus │    │   Loki   │
   │/Jaeger│  │ /Mimir     │    │ /Datadog │
   └───────┘  └────────────┘    └──────────┘

Your services emit OTLP. The collector fans out to your backends. Instrument once, swap backends without changing app code. That’s the architectural payoff.

The collector — your single point of control

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  batch:
    timeout: 5s
    send_batch_size: 8192

  memory_limiter:
    check_interval: 2s
    limit_mib: 1500

  attributes/scrub:
    actions:
      - key: http.url
        action: hash                        # don't leak query strings
      - key: db.statement
        action: delete                      # don't leak SQL parameters

  resource:
    attributes:
      - key: deployment.environment
        from_attribute: env
        action: insert

exporters:
  otlphttp/tempo:
    endpoint: https://tempo.example.com
  prometheus:
    endpoint: 0.0.0.0:8889
  otlphttp/loki:
    endpoint: https://loki.example.com/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/scrub, resource, batch]
      exporters: [otlphttp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, attributes/scrub, batch]
      exporters: [otlphttp/loki]

A few production details:

memory_limiter before batch. Stops the collector from OOMing under traffic spikes.
Scrub PII / sensitive fields at the collector. Easier to audit than per-app.
Insert resource attributes centrally — every span gets deployment.environment, service.region, etc.
Run the collector as a sidecar and a fleet aggregator. Sidecars do per-service tail sampling; aggregators do cross-service consolidation.

Python — FastAPI

uv add opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-httpx opentelemetry-instrumentation-asyncpg

Auto-instrumentation:

opentelemetry-bootstrap -a install        # one-shot install of relevant instrumentations

opentelemetry-instrument \
  --service_name my-api \
  --traces_exporter otlp \
  --metrics_exporter otlp \
  --logs_exporter otlp \
  --exporter_otlp_endpoint http://otel-collector:4317 \
  uvicorn app.main:app

That’s it. Every FastAPI request, every httpx call, every asyncpg query becomes a span with timing and attributes. Zero app-code changes.

For custom spans:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def process_order(order_id: int):
    with tracer.start_as_current_span("process_order", attributes={"order.id": order_id}):
        await charge_card(order_id)
        await ship(order_id)

The span auto-captures duration. Status is OK unless you record_exception(e) or set_status(Status(StatusCode.ERROR)) — do that on every catch.

For metrics:

from opentelemetry import metrics

meter = metrics.get_meter(__name__)
order_counter = meter.create_counter("orders_total", unit="1", description="Orders created")

order_counter.add(1, {"region": "ap-south-1", "tier": "pro"})

Go — net/http or chi

import (
  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
  sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer() (*sdktrace.TracerProvider, error) {
  exp, err := otlptracegrpc.New(ctx)
  if err != nil { return nil, err }
  tp := sdktrace.NewTracerProvider(
    sdktrace.WithBatcher(exp),
    sdktrace.WithResource(resource.NewWithAttributes(
      semconv.SchemaURL,
      semconv.ServiceName("my-api"),
      semconv.DeploymentEnvironment("prod"),
    )),
  )
  otel.SetTracerProvider(tp)
  return tp, nil
}

func main() {
  tp, _ := initTracer()
  defer tp.Shutdown(context.Background())

  mux := http.NewServeMux()
  mux.HandleFunc("/users", handleUsers)

  // Wrap with otelhttp for free request spans
  http.ListenAndServe(":8080", otelhttp.NewHandler(mux, "api"))
}

func handleUsers(w http.ResponseWriter, r *http.Request) {
  ctx, span := otel.Tracer("my-api").Start(r.Context(), "handleUsers")
  defer span.End()
  // ...
}

otelhttp wraps any http.Handler and propagates context. Use it as the outermost middleware. For outbound calls:

client := http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

Rust — Axum + tracing-opentelemetry

[dependencies]
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
tracing-opentelemetry = "0.27"
opentelemetry = "0.26"
opentelemetry_sdk = { version = "0.26", features = ["rt-tokio"] }
opentelemetry-otlp = { version = "0.26", features = ["grpc-tonic"] }

use opentelemetry::trace::TracerProvider as _;
use opentelemetry_otlp::WithExportConfig;
use tracing_subscriber::prelude::*;

fn init_telemetry() -> anyhow::Result<()> {
    let exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_tonic()
        .with_endpoint("http://otel-collector:4317")
        .build()?;

    let provider = opentelemetry_sdk::trace::TracerProvider::builder()
        .with_batch_exporter(exporter, opentelemetry_sdk::runtime::Tokio)
        .with_resource(opentelemetry_sdk::Resource::new(vec![
            opentelemetry::KeyValue::new("service.name", "my-api"),
        ]))
        .build();

    let tracer = provider.tracer("my-api");
    let otel = tracing_opentelemetry::layer().with_tracer(tracer);

    tracing_subscriber::registry()
        .with(tracing_subscriber::EnvFilter::from_default_env())
        .with(tracing_subscriber::fmt::layer().json())
        .with(otel)
        .init();

    Ok(())
}

Then on your handlers:

#[tracing::instrument(skip(db))]
async fn get_user(State(db): State<DB>, Path(id): Path<Uuid>) -> Result<Json<User>> {
    tracing::info!(?id, "fetching user");
    // ...
}

#[tracing::instrument] creates a span. Every tracing::info! inside it becomes a log event, structured, with the span’s attributes.

Context propagation — the magic

Across services, OTel uses W3C Trace Context: a single traceparent HTTP header.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │  └────── trace-id (16B) ─────┘ └── parent-id ──┘  │
              version                                          flags

When service A calls B over HTTP, A injects this header. B reads it; B’s spans become children of A’s. Result: a single trace from edge to DB across N hops.

Auto-instrumentation handles propagation for you. If you build your own transport (a queue, a webhook), you must propagate manually:

from opentelemetry.propagate import inject, extract

# Producer
headers = {}
inject(headers)
queue.publish(payload, headers=headers)

# Consumer
ctx = extract(headers)
with tracer.start_as_current_span("consume", context=ctx):
    process(payload)

Without this, your trace ends at the queue and resumes elsewhere with no link.

Sampling — the cost question

Tracing every request is expensive. Most teams sample.

Head-based (probabilistic)

Decide at the root whether to record. Cheap, simple, but you might miss rare errors.

processors:
  probabilistic_sampler:
    sampling_percentage: 10

10% of traces, decided at trace start. The decision propagates to every child span.

Tail-based

Decide after the trace completes. Keep all errors, slow traces, and a sample of normal ones.

processors:
  tail_sampling:
    decision_wait: 30s
    policies:
      - { name: errors,   type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow,     type: latency,     latency: { threshold_ms: 500 } }
      - { name: probably, type: probabilistic, probabilistic: { sampling_percentage: 5 } }

Tail-based gives you the best of both: full visibility on slow + error traces, low overhead on the rest. Configure on the collector (so your services don’t make decisions about cross-service traces).

Per-route or per-customer sampling

Sample more for important routes (/checkout) and less for noisy ones (/healthz):

- name: healthz
  type: string_attribute
  string_attribute: { key: http.target, values: [/healthz], invert_match: false }
  sampling: { sampling_percentage: 1 }

Logs — finally first-class

Pre-2025 OTel logs were experimental. They’re stable now and the missing leg is filled in:

Apps emit logs in OTel format.
The collector receives via OTLP.
Logs are exported to Loki/Datadog/whatever, with the trace_id and span_id automatically attached.

In Grafana, click any error log → jump straight to the trace it belongs to. That’s the workflow this enables.

SLOs, dashboards, RED/USE

Once you have OTel data flowing, build dashboards on the standard signals:

RED method (services): Rate, Errors, Duration.
USE method (resources): Utilization, Saturation, Errors.

Wire them to your SLOs (see SLOs and Error Budgets ). The OTel histograms make latency SLOs trivial:

histogram_quantile(0.95,
  sum(rate(http_server_request_duration_seconds_bucket{job="api"}[5m])) by (le)
)

Common mistakes

1. Too many attributes

A span with 200 attributes is useful to nobody and expensive everywhere. Stick to the semantic conventions for the basics. Add custom attributes sparingly.

2. High-cardinality attributes on metrics

user_id as a metric label is a memory bomb. Cardinality explosion at the metrics backend is the #1 OTel-related outage I’ve seen. Cardinality goes in traces (one value per request), not metrics.

3. No resource attributes

Spans without service.name, service.version, deployment.environment are useless when you’re investigating a bad deploy. Set them in the SDK init or in the collector resource processor.

4. Synchronous exports

SimpleSpanProcessor blocks on every span. Use BatchSpanProcessor everywhere except local debugging.

5. Forgetting to shut down

If your process exits before the batch flushes, you lose telemetry. Always:

provider.shutdown()                # at exit

defer tp.Shutdown(ctx)

provider.shutdown();

A realistic adoption path

Stand up the collector in dev. Configure receivers, basic processors, one exporter.
Auto-instrument one service. Get traces flowing.
Add request metrics (RED) on that service. Wire dashboards.
Convert logs to OTel format. Confirm trace correlation works.
Roll to a second service. Confirm cross-service traces work.
Sampling: head-based 10% to start, evolve to tail-based.
Repeat across the fleet. Standardize on the collector config.

Most teams spend a sprint on steps 1–4 and trickle through 5–7 over a quarter.

The OTel mental model#

Architecture#

The collector — your single point of control#

Python — FastAPI#

Go — net/http or chi#

Rust — Axum + tracing-opentelemetry#

Context propagation — the magic#

Sampling — the cost question#

Head-based (probabilistic)#

Tail-based#

Per-route or per-customer sampling#

Logs — finally first-class#

SLOs, dashboards, RED/USE#

Common mistakes#

1. Too many attributes#

2. High-cardinality attributes on metrics#

3. No resource attributes#

4. Synchronous exports#

5. Forgetting to shut down#

A realistic adoption path#

Read this next#