How to Run OpenTelemetry on a Small Cluster Without a Vendor Bill

Hosted observability is metered on what you send it, which means the meter is driven by span count, sample rate, active series and log volume rather than by how big your cluster is. A small cluster with chatty instrumentation can therefore produce a lot of billable telemetry. The self-hosted path moves that variable onto disks you own and configuration you control, and both of those are things you can size in advance.

The collector is a pipeline, and only pipelines run

A collector configuration has four kinds of pipeline component. Receivers take telemetry in, processors modify it on the way through, exporters send it out, and connectors join one pipeline to another by acting as an exporter on one side and a receiver on the other. Extensions sit beside all of that and provide capabilities that do not touch telemetry, like a health check endpoint.

The rule that catches everyone once is that defining a component does nothing. A receiver is enabled by being added to a pipeline under the service section, and the same is true of processors, exporters and connectors. A config file can look complete, load without complaint, and move no data at all.

Pipelines are typed as traces, metrics or logs, and every component in one has to support that type.

# /etc/otelcol/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: []

The exporter list is left empty on purpose, and the next section explains why. Note the endpoints as well. The collector itself defaults to localhost, and the documentation uses 0.0.0.0 for convenience while warning about it, so bind to the narrowest address that still lets your workloads reach it.

Name the exporter carefully

The OTLP exporters were renamed. otlp became otlp_grpc and otlphttp became otlp_http in collector release v1.50.0/v0.144.0, with the old names kept as deprecated aliases. Both spellings therefore work today and the old ones are on their way out, which means a config copied from a two year old blog post and a config copied from the current documentation will disagree with each other and both appear to be right.

Check the exporter README for the version you actually run, then validate before restarting anything.

otelcol validate --config=/etc/otelcol/config.yaml

Processor order is behaviour, not style

Processors run in the order you list them, and the collector repository publishes a recommended order rather than leaving it to taste. memory_limiter goes first so it can shed load before anything downstream has accumulated data it cannot flush. Sampling and filtering come next, so dropped data is never processed further. Processors that depend on request context, such as the Kubernetes attributes processor, have to run before batching, because batching clears that context. Enrichment follows. Batching goes last, and where the exporter offers its own batching, prefer that.

processors:
  memory_limiter:
    check_interval: 5s
    limit_mib: 4000
    spike_limit_mib: 500

Size those numbers against the memory limit of the container the collector runs in, not against the host.

Agent, gateway, or both

Two deployment patterns are documented. In the agent pattern the collector runs alongside the workload, as a sidecar or a DaemonSet, and applications point at it locally. In the gateway pattern applications and other collectors send to a single OTLP endpoint served by a standalone collector service, typically one per cluster or region.

The gateway buys separation of concerns, since credentials for the backends live in one place, and central policy such as filtering or sampling. It costs one more component to maintain and to fail, added latency when collectors are cascaded, and more resource usage overall.

On a small cluster, a DaemonSet agent for host and pod telemetry plus one gateway deployment is a sensible shape. If you scale the gateway to several replicas, respect the single writer principle for metrics, since concurrent writers reporting on the same resource can overwrite each other and degrade the data.

Where the data lands

Metrics can go straight into Prometheus, which accepts OTLP when started with --web.enable-otlp-receiver. The path is /api/v1/otlp/v1/metrics, so an SDK or exporter configured with a base URL of http://prometheus:9090/api/v1/otlp will append the signal path itself. Prometheus documents a few settings worth reading first, including otlp.translation_strategy for metric naming and storage.tsdb.out_of_order_time_window for out of order ingestion.

Traces fit Grafana Tempo, which accepts OTLP through the distributor on the standard 4317 and 4318 ports and stores blocks on local disk, S3, GCS or Azure.

Logs fit Grafana Loki, which exposes an OTLP endpoint at /otlp on the Loki address and expects the OTLP HTTP exporter.

How much disk a retention window needs

Prometheus publishes the sizing formula, so metrics can be estimated rather than guessed.

needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

It stores roughly one to two bytes per sample on average. Take 200,000 active series scraped every 30 seconds. That is 200,000 divided by 30, about 6,667 samples per second. Thirty days is 2,592,000 seconds, so the window holds about 17.3 billion samples. At two bytes each that is around 34.6 GB, and at one byte around 17.3 GB. Default retention is 15 days, set by --storage.tsdb.retention.time, and if you also set a size limit whichever triggers first wins.

Traces and logs have no equivalent per-item constant worth quoting, because a span with forty attributes and a span with four differ by an order of magnitude. Measure your own for a day, then multiply. What matters more is that both backends need retention set deliberately. Tempo keeps blocks for the compactor's block_retention, which defaults to 336h, or 14 days. Loki keeps logs indefinitely by default, since retention_period in limits_config defaults to 0s, and deletion only happens once the compactor has retention_enabled set to true with a delete_request_store configured.

That Loki default is the one that quietly fills a disk. It is also the easiest to fix, and fixing it before the first month of logs lands is considerably less work than fixing it afterwards.

What this leaves you with

Three storage components, one collector configuration, and a volume you can calculate from your own retention windows rather than read off an invoice. The work that remains is ordinary operations, which is upgrades, capacity and making sure the thing watching production is itself watched.

If you want that stood up and kept running on your cluster rather than assembled once and left to drift, that is our Kubernetes management and monitoring and observability work. For the dashboards and alerting layer that sits on top of the metrics side, see our guide to server monitoring with Prometheus and Grafana.