Task queues, pub/sub fan-out, event streaming, and request-reply are distinct problems. Choosing the tool by name before defining the problem produces migration costs and inflated monthly bills. Kafka, RabbitMQ, and SQS each solve them differently.

Kafka: a distributed log with replay

Kafka stores events in a distributed log that consumers can replay from the beginning. Consumer groups scale by adding members. Throughput reaches hundreds of thousands of messages per second on modest clusters. Operations demand dedication: brokers, partitions, rebalances, and retention need attention even on MSK or Confluent. Use Kafka for event sourcing, analytics pipelines, and broadcast to multiple independent consumers.

RabbitMQ: a smart broker for work queues

RabbitMQ filters by key patterns in topic exchanges, captures rejected messages in dead-letter exchanges, reorders deliveries by priority, and replicates state across nodes with quorum queues. It handles tens of thousands of messages per second with headroom and covers task queues with complex topology using fewer moving parts than a Kafka cluster.

SQS: managed queues inside AWS

SQS charges per message and scales on its own. Limits: 256KB per message, 15-minute maximum delay, pricing in cents per million requests. FIFO queues add strict ordering with throughput ceilings (300 req/s default, up to 3,000 with batching). AWS-native teams get queue infrastructure without managing servers.

Ordering by partition

Kafka guarantees order within each partition. The partition key decides the sequence: orders partitioned by customer_id preserve per-customer order. RabbitMQ also drops global ordering under competing consumers. Design idempotent consumers regardless of the tool. Infrastructure ordering optimizes, idempotency corrects.

Operational cost

  • Managed Kafka (MSK, Confluent): smallest cluster starts near $200/month
  • Self-hosted RabbitMQ: cheap instance, but HA, failover, and upgrades land on your backlog
  • SQS: pay-per-message pricing, scales to zero when traffic disappears

Where to start

Start with SQS when the team lives in AWS, or RabbitMQ when rich routing matters outside it. Move to Kafka when history replay, multiple independent consumers, or 100k+ msg/s become real requirements. A small team running a job queue needs a reliable queue with decent observability, not its own cluster.