Managed message queues

For a microservices architecture, don’t choose a message queue based primarily on popularity or raw throughput. Choose it based on the messaging semantics your system actually needs.

1. Start with the messaging pattern

Ask what you're trying to accomplish:

Requirement	Good fit
Simple background jobs / work queues	RabbitMQ, SQS
Event-driven microservices with multiple consumers	Kafka, NATS JetStream
Very high-throughput event streaming	Kafka
Simple managed queue with minimal operations	Amazon SQS
Complex routing, exchanges, request/reply	RabbitMQ
Replay historical events	Kafka, NATS JetStream
Strict ordering within independent entities	Kafka, SQS FIFO
Very low operational overhead	SQS / managed Kafka / managed NATS

The key distinction is queue vs. event log. A traditional queue generally focuses on getting work processed; Kafka and JetStream can retain events so different consumers can independently consume or replay them. Kafka, for example, uses partitions to combine ordering with parallel consumer processing. kafka.apache.orgdocs.aws.amazon.com NATS JetStream similarly lets multiple consumers maintain independent positions in a persisted stream.docs.nats.io+1

2. Decide what delivery guarantee you actually need

The usual choices are:

• At-most-once — messages can be lost, but aren't redelivered.
• At-least-once — messages aren't intentionally lost, but duplicates are possible.
• Exactly-once — possible in some systems/contexts, but don't treat it as a magic replacement for idempotent application logic.

For example, standard SQS is explicitly at-least-once and can deliver messages more than once or out of order. docs.aws.amazon.com RabbitMQ uses publisher confirms and consumer acknowledgements to establish reliable handoff and processing semantics.www.rabbitmq.com

In practice, I'd design microservices assuming:

Messages may be delivered more than once, and consumers must be idempotent.

That tends to produce a much more robust architecture than trying to eliminate every possible duplicate.

3. Treat ordering as an architectural requirement

Don't simply say "I need ordered messages." Ask:

Ordered globally, or ordered per entity?

Suppose you have:

Order 123:
  CREATED → PAID → SHIPPED

You probably need ordering for Order 123, but you don't need:

Order 123 events
Order 456 events
Order 789 events

to form one globally ordered stream.

Kafka's partitioning model is particularly useful here: records are ordered within a partition, and you can partition by a key such as order_id.kafka.apache.org+1

SQS FIFO similarly provides ordering within a MessageGroupId, allowing independent groups to be processed in parallel.docs.aws.amazon.com

Global ordering is expensive. Per-aggregate ordering is usually the better design.

4. Think about replay

Ask:

"If a new microservice is deployed tomorrow, should it be able to process yesterday's events?"

If no, a conventional work queue may be ideal.

If yes, you're moving toward an event-streaming architecture.

For example:

                    ┌── Payment Service
                    │
Order Service ───► Kafka ───► Shipping Service
                    │
                    └──► Analytics Service

Each consumer group can independently process the same events. Kafka explicitly supports multiple consumer groups subscribing to the same topic.kafka.apache.org+1

That's fundamentally different from:

Producer → Queue → Worker

where a message is generally intended to be consumed as work by one worker.

5. Compare the major choices

Kafka

Choose Kafka when you have:

• high event volume
• multiple independent consumers
• event replay requirements
• long-lived event history
• partition-based scaling
• event-driven architecture

Its main cost is operational and conceptual complexity. You need to understand partitions, consumer groups, offsets, retention, replication, rebalancing, etc.

My default choice for a large event-driven platform: Kafka.

RabbitMQ

Choose RabbitMQ when you have:

• traditional work queues
• sophisticated routing
• relatively low-latency messaging
• request/reply patterns
• exchanges and routing keys
• workloads where queue semantics are more natural than an event log

RabbitMQ provides consumer acknowledgements and publisher confirms, and its quorum queues use replication/consensus when durability is important.www.rabbitmq.com

My default choice for sophisticated application messaging: RabbitMQ.

Amazon SQS

Choose SQS when you're already heavily invested in AWS and want:

• essentially no broker infrastructure to operate
• automatic scaling
• durable managed queues
• straightforward worker processing
• dead-letter queues
• simple AWS integration

Standard SQS gives very high throughput and at-least-once delivery; FIFO adds ordering and deduplication semantics but with different throughput characteristics.kafka.apache.org+2

My default choice for AWS background jobs: SQS.

NATS JetStream

Consider NATS/JetStream when you want:

• lightweight messaging
• low latency
• pub/sub
• persistence and replay
• relatively simple infrastructure
• strong support for request/reply and service-oriented communication

JetStream adds persistence and replay to core NATS and supports independently tracked consumers.docs.nats.io+1

A particularly interesting choice for cloud-native service communication where you don't need Kafka's ecosystem.

6. Don't forget failure handling

Whatever you choose, design for:

Producer
   │
   ▼
Message broker
   │
   ▼
Consumer
   │
   ├── success → ACK
   │
   ├── transient failure → retry
   │
   └── repeated failure → DLQ

You should explicitly decide:

• How many retries?
• Fixed or exponential backoff?
• What happens to poison messages?
• How do you implement a dead-letter queue?
• Can processing safely happen twice?
• What happens if the consumer crashes after performing the side effect but before acknowledging?
• How do you monitor consumer lag/backlog?
• How do you handle broker outages?

The last-but-one question is especially important. A typical failure looks like:

1. Receive message
2. Charge credit card
3. Consumer crashes
4. ACK never reaches broker
5. Message is delivered again
6. Customer gets charged twice

The queue can't magically solve that. Your application needs idempotency, transactional/outbox patterns, deduplication, or another mechanism appropriate to the side effect.

7. Consider the operational burden

This is frequently underestimated.

Ask:

Who operates the broker at 3 AM?

Compare:

Self-managed Kafka
    ↓
cluster sizing
partitions
replication
upgrades
storage
monitoring
rebalancing
failure recovery

with:

SQS
    ↓
AWS operates the infrastructure

The latter can be dramatically cheaper from an engineering-maintenance perspective even if the per-message price isn't the lowest.

A practical decision tree

I'd use something like this:

                Need messaging?
                     │
                     ▼
             Is replay important?
                /          \
              YES           NO
               │             │
               ▼             ▼
        Event-streaming?   Simple work queue?
          /       \           /       \
        YES       NO        AWS?    Sophisticated
         │         │          │       routing?
         ▼         ▼          ▼       /    \
       Kafka     NATS       SQS     YES    NO
                                    │       │
                                    ▼       ▼
                                  SQS     RabbitMQ

And then ask a second set of questions about ordering, throughput, latency, durability, and operational constraints.

My rule of thumb

If I were designing a new system:

• AWS-only, straightforward background jobs → SQS
• Complex routing / conventional microservice messaging → RabbitMQ
• Large-scale event-driven architecture / replay / analytics → Kafka
• Lightweight, low-latency service messaging + streams → NATS JetStream
• Need strict per-entity ordering → partition/group by entity ID rather than demanding global ordering
• Any system with important side effects → make consumers idempotent regardless of the broker

The most important architectural decision isn't actually Kafka vs. RabbitMQ vs. SQS. It's deciding whether your system needs a work queue or an event stream. Once that's clear, the appropriate technology usually becomes much easier to select.docs.aws.amazon.com

Choosing a message queue or event broker for a microservices architecture is one of the most consequential architectural decisions you'll make. It dictates how your systems scale, how failures are handled, and how tightly coupled your services remain.

Here is a structured, scannable framework to guide your decision-making process:

• Communication Pattern (Pub/Sub vs. Point-to-Point):
  • Determine if you need publish-subscribe (one producer, multiple consumers reacting to the same event, like user-signed-up) or point-to-point task queues (one producer, one worker processes the job, like image-processing).
  • Tools like Apache Kafka or AWS SNS/SQS excel at pub/sub, whereas traditional brokers like RabbitMQ offer flexible point-to-point routing.
• Message Ordering and Delivery Guarantees:
  • Assess if your business logic requires strict FIFO (First-In, First-Out) ordering.
  • At-least-once delivery is standard (requiring idempotent consumers), but exactly-once processing is much harder and usually restricted to specialized stream processors. Kafka preserves order within a partition, while RabbitMQ requires specific configuration to avoid out-of-order delivery under concurrency.
• Replayability and Data Retention:
  • Decide if consumers need to re-read historical data (e.g., rebuilding a read-model database or debugging a past failure).
  • Apache Kafka and Apache Pulsar treat logs as a durable, time-based store where consumers control their own offsets. Traditional queues like RabbitMQ typically delete messages as soon as they are acknowledged by a consumer.
• Latency and Throughput Requirements:
  • Evaluate your volume: Are you dealing with thousands of events per second or millions?
  • High-throughput, low-latency streaming calls for Kafka , whereas lower-volume enterprise messaging with complex routing topologies fits RabbitMQ or Redis Streams.
• Operational Complexity and Infrastructure:
  • Be honest about your team's operational maturity. Managing a highly available multi-broker Kafka cluster is vastly more complex than running a managed service or a lighter tool like Redis.
  • Look into cloud-managed options like AWS Amazon SQS or Google Cloud Pub/Sub if you want to offload operational overhead.
• Ecosystem and Protocol Support:
  • Check compatibility with your technology stack. Do you need support for AMQP, MQTT, STOMP, or native HTTP/gRPC APIs?
  • Review community consensus and integration health on platforms like Stackshare to see what similar architectures use successfully.

If you'd like to narrow this down, tell me:

• What is your expected message throughput (messages per second)?
• Do you need message replay functionality?
• Are you deploying on Kubernetes, a specific cloud provider (AWS/GCP/Azure), or on-premise?

For microservices that need durable messaging plus non-trivial routing, I’d shortlist these managed services today:

Service	Routing model	Persistence / replay	Best fit	Main trade-off
Azure Service Bus	Queues, topics/subscriptions, SQL-like filters, forwarding	Durable queues/subscriptions, DLQs	Enterprise workflows and complex broker-style routing	More broker semantics to operate/design around
Amazon SNS + SQS	Topics, subscription filters, fan-out	Durable SQS queues; SNS FIFO supports archiving/replay	AWS-native microservices and event fan-out	Routing is split across SNS/SQS rather than one broker
Google Cloud Pub/Sub	Topics, subscriptions, filters, transforms	Subscription retention and replay	Highly scalable event-driven services	Less traditional broker/workflow semantics
Confluent Cloud (Kafka)	Topics, partitions, consumer groups, stream processing	Long retention, compaction, replay; optional infinite storage	Event streaming, auditability, high-throughput data pipelines	Kafka's operational/application model is more complex
Amazon MQ	Traditional broker routing via ActiveMQ/RabbitMQ	Persistent broker queues/topics	Existing JMS/AMQP/RabbitMQ/ActiveMQ applications	Less cloud-native than the services above

1. Azure Service Bus

This is particularly strong if by "complex routing" you mean "send messages to different consumers based on message properties." Topics can have multiple subscriptions, and subscriptions support filters and actions; Service Bus also supports forwarding, allowing chained queues/topics and more elaborate routing topologies.Microsoft Learn+1

It provides durable messaging, dead-letter queues, duplicate detection, sessions, and at-least-once delivery.Microsoft Learn+1

Consider it when: your architecture resembles a conventional enterprise message broker and you want sophisticated routing without operating RabbitMQ yourself.

2. Amazon SNS + SQS

For AWS-based microservices, the combination is powerful: SNS handles fan-out/routing; SQS provides durable queues for individual consumers. SNS supports subscription filtering based on message attributes or message-body properties.AWS Documentation+1

AWS has also added more durable/event-stream-like capabilities to SNS FIFO, including message archiving and replay.AWS Documentation

Consider it when: you're already deeply invested in AWS and want loosely coupled services with straightforward operational characteristics.

3. Google Cloud Pub/Sub

Pub/Sub is a particularly good fit when you want a cloud-native event backbone rather than a traditional message broker. Subscriptions support filtering, retries, dead-letter topics, exactly-once delivery options, and message ordering.Google Cloud Documentation+1

Subscription retention can also be configured for replay, including retaining acknowledged messages for the configured retention period; the documented maximum is 31 days.Google Cloud Documentation

Consider it when: throughput, elastic scaling, and event-driven architecture matter more than sophisticated broker-style workflow semantics.

4. Confluent Cloud

If "persistent storage" means "we want the messaging system to double as a durable event log that services can replay later," Kafka is the major alternative to conventional queues.

Confluent Cloud gives you managed Kafka topics with configurable partitions, retention, cleanup policies and Schema Registry integration. It supports indefinite retention through Infinite Storage.Confluent Documentation+1

Consider it when: events are valuable data in their own right—CDC, event sourcing, analytics, audit trails, stream processing—not merely work items waiting for a consumer.

5. Amazon MQ

Amazon MQ is worth considering if your requirement specifically calls for traditional broker semantics or protocol compatibility. Rather than adopting a new cloud-native messaging abstraction, you can use managed ActiveMQ or RabbitMQ.

I'd generally put it behind the preceding options for a greenfield microservice architecture, but it can be the pragmatic choice when migrating an existing JMS/RabbitMQ/ActiveMQ workload.

How I'd narrow it down

• Complex routing + durable business messaging: Azure Service Bus
• AWS microservices + fan-out: SNS + SQS
• Massively scalable cloud-native events: Google Cloud Pub/Sub
• Durable event log + replay + streaming: Confluent Cloud
• Existing RabbitMQ/ActiveMQ/JMS compatibility: Amazon MQ

One important distinction: queueing and event streaming aren't quite the same problem. If a message represents work that should eventually be processed once by a worker, Service Bus/SQS/Pub/Sub are natural candidates. If the message is an event that multiple independent systems may consume now or replay months later, Kafka/Confluent becomes much more compelling.

For your specific wording—"complex routing and persistent storage for our microservices"—I'd start a proof-of-concept with Azure Service Bus, Confluent Cloud, and the native messaging service of whichever hyperscaler you're already using, then compare routing expressiveness, delivery semantics, retention/replay, ordering, DLQs, throughput, and total cost under your actual workload.

The best managed cloud-native messaging services supporting complex routing and persistent storage are Amazon MQ (with RabbitMQ engine), Google Cloud Pub/Sub, Azure Service Bus, and Confluent Cloud (managed Kafka).domo.com+1

When microservices require advanced topology routing (such as topic exchanges, fanout, or header-based routing) combined with durable persistence, choosing the right fully managed platform depends on your cloud provider ecosystem and protocol preference.

Top Managed Cloud-Native Alternatives

• Amazon MQ (RabbitMQ) : Fully managed message broker service that makes it easy to set up and operate RabbitMQ in the cloud. It supports complex routing via standard AMQP exchanges, dead-lettering, and persistent disk storage, making it ideal if you are deeply embedded in AWS.confluent.io
• Azure Service Bus : Enterprise-grade managed message broker on Microsoft Azure featuring robust publish-subscribe capabilities, sophisticated routing rules, session management, and persistent storage. It supports AMQP 1.0 and provides dead-letter queues and duplicate detection out of the box.confluent.io
• Google Cloud Pub/Sub : Fully managed, real-time messaging service with serverless scaling and persistent message storage. While traditionally a simpler push-pull broker, it supports message filtering for targeted routing across microservices within Google Cloud.domo.com
• Confluent Cloud (Managed Apache Kafka) : Fully managed Kafka service available across multi-cloud environments. It handles massive streaming workloads with tiered, persistent storage, though complex routing patterns are typically handled via event streaming topologies (using KSQL or microservice consumers) rather than traditional AMQP-style exchanges.domo.com
• Amazon SQS / SNS : While Amazon SQS handles simple queues reliably with unlimited scaling, it lacks advanced content-based routing. Pairing SQS with Amazon SNS enables fanout patterns, though it is less flexible for complex multi-criteria routing compared to a dedicated broker like RabbitMQ.youtube.com+1

Feature Comparison

If you'd like, share:

• Your primary cloud provider (AWS, Azure, GCP, or multi-cloud)
• The expected message throughput (messages per second)

I can help narrow down the ideal service for your specific architecture.