Log pipeline for cost control and long retention

Which brands AI mentions

(Best default for low cost)

Loki indexes log labels rather than the full text of every log line, reducing index overhead. It stores compressed log chunks in object storage such as S3 or GCS.grafana.com+1

• Pipeline: OpenTelemetry Collector or Fluent Bit → Loki → S3/GCS.
• Retention: Set retention policies by stream or tenant.
• Trade-off: Full-text searches across large historical windows can be less convenient than in a full-text search engine.

Best for Kubernetes, infrastructure logs, and Grafana-centric observability.

2. OpenSearch + S3 tiers

(Best for searchable history)

Keep recent logs on fast storage and transition older indexes to more economical warm storage. For infrequently accessed archives, S3-based direct querying can avoid indexing all historical data.docs.aws.amazon.com+1

• Pipeline: Collector → OpenSearch → warm storage / S3 archive.
• Retention: Automate transitions and deletion with lifecycle policies.
• Trade-off: Indexing, compute, and cluster operations can cost significantly more than simply archiving logs.

Best when engineers need full-text search, structured queries, and dashboards over historical events.

3. ClickHouse + object storage

(Best for high-volume analytics)

ClickHouse supports compressed columnar storage, TTL-based data management, and storage tiers backed by S3. Older data can use cheaper storage while retaining analytical query capabilities.github.com+1

• Pipeline: OpenTelemetry Collector → ClickHouse → local SSD / S3-backed storage.
• Retention: Use TTLs to move, recompress, or remove data.
• Trade-off: Requires schema and query design; operational complexity depends on deployment model.

Best for high ingestion volumes, SQL-based investigations, and aggregations over months of data.

My recommended design for cost control

Collection and filtering

OpenTelemetry Collector or Fluent Bit

Parse · redact · filter · sample noisy debug logs

Hot searchable logs

Loki or OpenSearch

7–30 days as an illustrative starting point

Long-term archive

S3 / compatible object storage

90 days to several years, based on requirements

Lifecycle, security and cost controls

Retention enforcement · encryption · access controls · budget alerts

The key cost-saving principle is to avoid paying to fully index every log for its entire retention period. Keep a short, fast search window; archive raw logs cheaply; and use a query or rehydration path for older data. If you need searchable history without a restore workflow, compare Loki and ClickHouse against OpenSearch's warm-storage or direct-query options.

One important detail: make sure the archive can actually be queried or restored within your incident-response requirements. Object storage alone is cheap, but it isn't equivalent to an immediately searchable log index.

Let's narrow down the best fit

1. How much log data do you ingest?

Under 50 GB/day

50–100 GB/day

100–500 GB/day

Over 500 GB/day

2. How long must you retain logs?

30–90 days

6–12 months

1–3 years

5–7+ years

3. Where does the pipeline run?

AWS

Google Cloud

Azure

On-premises / multi-cloud

4. What does historical access need to look like?

Search any retained log on demand

Cheap archive; restore before searching

Compliance retention, rarely queried

Fast SQL analytics over historical data

5. Operational preference

Managed service preferred

Self-hosted open source

Lowest total cost, even with more operations

[Compare architectures for my workload]