Fixing AWS SQS Message Visibility Timeouts That Cause Duplicate Processing

Amazon Simple Queue Service (SQS) is one of the most widely used messaging services for building distributed systems.

It powers:

• Background workers
• Order processing
• Email delivery
• Payment workflows
• Image processing
• Event-driven architectures
• Microservices

A typical workflow looks like:

Application
      │
      ▼
    SQS Queue
      │
      ▼
Worker Service
      │
      ▼
Database/API

Developers often assume:

One Message
↓
One Processing

In reality, Amazon SQS provides:

At-Least-Once Delivery

This means a single message may be delivered more than once.

One of the biggest reasons is an incorrectly configured:

Visibility Timeout

The consequences can be severe:

• Duplicate orders
• Double payments
• Multiple emails
• Repeated invoices
• Inventory inconsistencies
• Duplicate database records

The queue itself is usually behaving correctly.

The problem lies in how workers process messages.

In this guide, you'll learn why visibility timeouts exist, how duplicate processing occurs, and how to design reliable consumers that remain correct even when messages are delivered multiple times.

What You Will Learn From This Article

After reading this guide, you'll understand:

• How SQS visibility timeouts work.
• Why duplicate processing occurs.
• The message lifecycle.
• How long-running jobs affect queues.
• Dynamic visibility extensions.
• Idempotent processing.
• Production best practices.

Understanding Message Visibility

When a worker receives a message:

Message
↓
Delivered

it is not immediately removed from the queue.

Instead:

Message
↓
Hidden
Temporarily

This hidden period is called the:

Visibility Timeout

Why Visibility Exists

Imagine two workers:

Worker A

Worker B

Without visibility:

Both Workers
Process
Same Message

Visibility temporarily prevents this.

Standard Workflow

Receive Message
      │
      ▼
Message Hidden
      │
      ▼
Worker Processes
      │
      ▼
Delete Message

Everything works correctly.

The Problem Begins

Suppose:

Visibility timeout:

30 Seconds

Processing time:

45 Seconds

Timeline:

0s
Receive Message

30s
Visibility Expires

31s
Second Worker Receives Message

45s
First Worker Finishes

Now:

Same Message
Processed Twice

Why This Happens

The first worker never deleted the message before:

Visibility Timeout

expired.

SQS assumes:

Worker Failed

and makes the message available again.

This is expected behavior.

Common Symptoms

You may notice:

Duplicate Emails

Duplicate Orders

Repeated Billing

Multiple API Calls

Inventory Errors

Duplicate Database Rows

These often indicate visibility timeout problems.

Default Visibility Timeout

Many queues start with:

30 Seconds

For lightweight workloads this is sufficient.

For video processing, backups, or large data imports, it is often too short.

Long-Running Jobs

Imagine processing:

2 GB Video

Time required:

5 Minutes

Visibility timeout:

30 Seconds

Result:

Repeated Processing

The queue behaves correctly.

The configuration does not.

Solution 1

Increase Visibility Timeout

Configure the queue to exceed expected processing time.

Example:

Expected Job
=
2 Minutes

Visibility
=
5 Minutes

This provides a safety margin.

Solution 2

Extend Visibility Dynamically

Sometimes processing time is unpredictable.

SQS supports:

ChangeMessageVisibility

Workflow:

Receive Message
      │
      ▼
Start Processing
      │
      ▼
Extend Visibility
      │
      ▼
Continue Processing

This prevents premature redelivery.

Don't Make Visibility Too Long

Some developers set:

12 Hours

for every queue.

This creates another problem.

If the worker crashes:

Message Hidden
For Hours

Recovery becomes unnecessarily slow.

Choose a value that reflects realistic processing times.

Delete Messages Promptly

Successful processing should end with:

DeleteMessage

Do not delay deletion after the work has completed.

Understand At-Least-Once Delivery

Even with perfect visibility settings:

Duplicate Messages

may still occur.

SQS Standard queues guarantee:

At Least Once

not:

Exactly Once

Applications must be designed accordingly.

Build Idempotent Consumers

An idempotent operation produces the same result regardless of how many times it is executed.

Example:

Bad:

Charge Credit Card

every time a message arrives.

Better:

Check Payment ID
↓
Already Processed?
↓
Skip Duplicate

This prevents duplicate business actions.

Store Processing IDs

Maintain a table containing:

• Message ID
• Business ID
• Processing timestamp

Before processing:

Already Exists?

If yes:

Ignore Duplicate

Simple deduplication dramatically improves reliability.

Dead Letter Queues

Messages that repeatedly fail should eventually move to:

Dead Letter Queue

This prevents endless retry cycles.

Configure:

• Maximum receive count
• DLQ monitoring
• Alerting

Monitor Queue Metrics

Useful CloudWatch metrics include:

• Approximate Age of Oldest Message
• Messages Received
• Messages Deleted
• Messages Visible
• Messages Not Visible

Unexpected trends often reveal timeout issues.

Real-World Example

An image-processing service handles:

Large RAW Images

Each job requires:

90 Seconds

Queue configuration:

Visibility Timeout
=
30 Seconds

After:

30 Seconds

another worker receives the same message.

Both workers generate thumbnails.

Customers receive duplicate notifications.

The solution:

• Increase visibility timeout
• Extend visibility during processing
• Implement idempotent job tracking

Duplicate processing disappears.

FIFO Queues

Amazon SQS FIFO queues provide additional guarantees:

• Ordered delivery
• Message deduplication

However:

visibility timeouts still matter.

Workers should still be idempotent.

Best Practices Checklist

When using Amazon SQS:

✅ Estimate realistic processing times

✅ Configure visibility timeout appropriately

✅ Extend visibility for long-running jobs

✅ Delete messages immediately after success

✅ Implement idempotent consumers

✅ Store processed message identifiers

✅ Configure Dead Letter Queues

✅ Monitor CloudWatch metrics

✅ Test worker crashes

✅ Simulate duplicate deliveries

Common Mistakes to Avoid

Avoid:

❌ Assuming messages are delivered exactly once

❌ Setting visibility timeout too short

❌ Setting visibility timeout excessively long

❌ Forgetting to delete processed messages

❌ Ignoring duplicate processing

❌ Skipping idempotency

❌ Failing to configure DLQs

Performance Considerations

As throughput increases:

• Worker concurrency grows.
• Processing times become more variable.
• Network delays fluctuate.
• Retry frequency increases.

Visibility timeout tuning becomes increasingly important at scale.

Periodic review of queue metrics helps maintain optimal performance.

Why This Bug Is Difficult to Diagnose

Many developers observe:

Duplicate Orders

and immediately investigate:

• Database bugs
• API retries
• Load balancers

In reality:

Queue Configuration

is often the root cause.

Because duplicate delivery is expected behavior in Standard queues, applications—not the queue—must ensure business operations remain correct.

Wrapping Summary

Amazon SQS visibility timeouts are a critical part of reliable distributed messaging. They temporarily hide messages while workers process them, preventing immediate duplicate delivery. However, when processing takes longer than the configured visibility timeout, SQS assumes the worker has failed and makes the message available again, leading to duplicate processing.

The solution is not simply increasing the timeout indefinitely. Production systems should configure realistic visibility periods, extend visibility dynamically for long-running tasks, delete messages immediately after successful processing, and most importantly, design idempotent consumers that can safely handle duplicate deliveries. Even with well-tuned queues, at-least-once delivery remains a fundamental characteristic of Amazon SQS Standard queues.

By combining proper timeout configuration, robust monitoring, dead-letter queues, and idempotent application logic, you can build messaging systems that remain reliable, scalable, and resilient even under heavy production workloads.