Sooner or later, every job runs twice. The worker dies after processing and before acking, the visibility timeout expires mid-job, a network partition splits the truth across two nodes. A queue system designed around single execution accumulates double charges, repeated emails, and corrupted stock. Design every handler for at-least-twice execution.

How do you guarantee idempotency in practice?

Three techniques cover almost every case. Natural keys dedupe by user, action, and window: UNIQUE(user_id, action, DATE(created_at)) blocks the second processing of the same event within the day. Idempotency tokens stored under unique constraints turn retries into no-ops; the second insert fails and the handler returns success. Conditional updates return affected row counts: zero rows means another execution already did the work.

Visibility timeout: how much is enough?

Starting point: p99 job duration times two. A timeout set too short spawns duplicates in bulk; too long delays retries of jobs already lost. Each tool implements the mechanism its own way: BullMQ renews locks, SQS exposes the native VisibilityTimeout parameter, Sidekiq ships reliable fetch with super_fetch. Monitor jobs that overrun the timeout; they are the source of your duplicates.

Retries without a thundering herd

Exponential backoff with jitter spreads attempts out: 1s, 4s, 16s with random noise keeps a thousand workers from hammering the external service in the same second. Classify errors before scheduling: 5xx and timeouts are retryable; rejected validation is permanent and heads straight to the dead-letter queue. Exhausted max attempts feed the DLQ with the original payload, ready for manual replay during the incident.

Queue hierarchy

A single queue mixes payment capture with report exports, and a bulk backlog starves critical work. Split three tiers with dedicated workers: critical (payments, fraud checks), default (normal flow), and bulk (emails, exports). The 200-thousand-row export stops delaying the capture webhook.

Observability and recurring jobs

  • Queue depth, wait time p95, processing time p95, and failure rate per job class
  • Alerts fire on depth growth trends; a high absolute value can mean routine peak-hour traffic

Recurring work needs a distributed lock: Redis SETNX with TTL or a Postgres advisory lock (pg_advisory_lock). Without the lock, two schedulers double-fire the same cron, and the idempotency token becomes the last line of defense. Add the lock; it costs one line and prevents the midnight ticket.