Queuing and messaging concepts (for example, publish/subscribe)

Task Statement 2.1: Design scalable and loosely coupled architectures.

📘AWS Certified Solutions Architect – (SAA-C03)


1. Introduction to Queuing and Messaging

Modern cloud applications are often built using loosely coupled architectures. In a loosely coupled system, different components of the application can work independently from each other.

Queuing and messaging systems help achieve this independence by allowing application components to communicate asynchronously.

Instead of one service directly calling another service and waiting for a response, the first service can send a message to a queue or messaging system. The receiving service processes the message later.

This approach improves:

  • Scalability
  • Reliability
  • Fault tolerance
  • System flexibility

In AWS architectures, the most commonly used messaging services are:

  • Amazon Simple Queue Service (SQS) – message queues
  • Amazon Simple Notification Service (SNS) – publish/subscribe messaging

2. What is a Message?

A message is a small piece of data sent from one component of an application to another.

A message usually contains:

  • Information about an event
  • Data that needs processing
  • Instructions for another service

Example in an IT system:

A web application uploads images.
After upload, the application sends a message containing:

  • Image file location
  • User ID
  • Processing instructions

A separate service later reads this message and performs image processing.


3. What is a Queue?

A queue is a temporary storage location for messages.

Messages are stored in the queue until a service retrieves and processes them.

The queue acts as a buffer between services.

How a Queue Works

  1. A producer sends a message to the queue.
  2. The message stays in the queue.
  3. A consumer retrieves the message.
  4. The consumer processes the message.
  5. The message is removed from the queue.

This allows the producer and consumer to operate independently.

Benefits of Queues

Queues help systems become:

1. Decoupled

Services do not depend directly on each other.

2. Scalable

More consumers can be added to process messages faster.

3. Fault tolerant

If the consumer service fails, the message remains in the queue.

4. Reliable

Messages are stored until processed.


4. Amazon Simple Queue Service (SQS)

Amazon SQS is a fully managed message queuing service that allows applications to send, store, and receive messages.

Key characteristics:

  • Fully managed
  • Highly scalable
  • Highly available
  • Supports asynchronous communication

SQS allows application components to communicate without needing to know each other’s status or availability.


5. Types of Amazon SQS Queues

There are two types of SQS queues.

1. Standard Queue

A Standard Queue provides:

  • Very high throughput
  • Best-effort message ordering
  • At-least-once message delivery

Characteristics

  • Messages may be delivered more than once
  • Message order is not guaranteed
  • Very high scalability

Use Cases

Standard queues are suitable for:

  • Large distributed systems
  • High message throughput systems
  • Event processing architectures

Example IT scenario:

A logging system sends application logs into a queue. Multiple processing services analyze logs in parallel.


2. FIFO Queue (First-In-First-Out)

A FIFO queue ensures:

  • Messages are processed in the exact order they are sent
  • Messages are delivered exactly once

Characteristics

  • Strict message ordering
  • No duplicate messages
  • Lower throughput than Standard queues

Use Cases

FIFO queues are useful when:

  • Order of operations matters
  • Duplicate processing must be prevented

Example IT scenario:

A payment processing system must process transactions in the exact order received.


6. SQS Message Lifecycle

Understanding the message lifecycle is important for the exam.

Step 1 — Send Message

A producer sends a message to the queue.

Step 2 — Store Message

The message is stored redundantly across multiple servers.

Step 3 — Receive Message

A consumer retrieves the message from the queue.

Step 4 — Visibility Timeout

When a consumer receives a message, it becomes temporarily invisible to other consumers.

This is called the visibility timeout.

Purpose:

  • Prevent multiple services from processing the same message simultaneously.

Step 5 — Delete Message

After successful processing, the consumer deletes the message.

If the message is not deleted before the visibility timeout expires, it becomes visible again.


7. Dead-Letter Queues (DLQ)

Sometimes messages cannot be processed successfully.

Examples:

  • Invalid data
  • Application errors
  • System failures

Instead of retrying forever, messages can be sent to a Dead-Letter Queue (DLQ).

A DLQ stores messages that fail processing multiple times.

Benefits:

  • Helps identify problematic messages
  • Prevents queue processing from being blocked
  • Allows developers to debug errors

8. Long Polling vs Short Polling

Consumers retrieve messages from SQS using polling.

Short Polling

The consumer immediately checks the queue for messages.

If no messages exist, the request returns empty.

This can lead to many unnecessary requests.


Long Polling

The consumer waits for messages to arrive.

Benefits:

  • Reduces empty responses
  • Reduces cost
  • Improves efficiency

AWS recommends using long polling.


9. What is Publish/Subscribe Messaging?

The Publish/Subscribe (Pub/Sub) model is a messaging pattern where:

  • Publishers send messages
  • Subscribers receive messages

Publishers do not know who the subscribers are.

Subscribers automatically receive messages they are subscribed to.

This model is useful for event-driven architectures.


10. Amazon Simple Notification Service (SNS)

Amazon SNS is a fully managed publish/subscribe messaging service.

SNS allows applications to send messages to multiple subscribers simultaneously.

SNS uses a central component called a Topic.


11. SNS Topic

An SNS topic is a communication channel used to send messages to subscribers.

Workflow:

  1. A publisher sends a message to the topic.
  2. The topic distributes the message to all subscribers.

12. SNS Subscribers

Subscribers can be many different AWS services or endpoints.

Examples include:

  • SQS queues
  • AWS Lambda
  • HTTP endpoints
  • Email notifications
  • SMS messages

This allows SNS to distribute events across multiple systems.

Example IT scenario:

A deployment system publishes an application deployment event.

Subscribers may include:

  • Logging service
  • Monitoring system
  • Notification service
  • Configuration update service

All subscribers receive the same event.


13. SNS Fan-Out Pattern

A common architecture pattern is SNS Fan-Out.

In this pattern:

  1. A message is sent to an SNS topic.
  2. The topic sends the message to multiple SQS queues.
  3. Each queue is processed by different services.

Benefits:

  • Multiple systems receive the same event
  • Services remain independent
  • Workloads are distributed

Example IT scenario:

A data ingestion service publishes a new data uploaded event.

SNS sends the event to multiple queues:

  • Data analytics queue
  • Data validation queue
  • Data backup queue

Each system processes the event independently.


14. Comparing SQS and SNS

FeatureAmazon SQSAmazon SNS
Messaging ModelQueuePublish/Subscribe
Message DeliveryOne consumer receives messageMultiple subscribers receive message
CommunicationPoint-to-pointOne-to-many
Message StorageStores messages until processedDoes not store long-term messages

15. Combining SNS and SQS

Many AWS architectures combine SNS and SQS.

Example architecture:

  1. A service publishes an event to SNS.
  2. SNS sends the message to multiple SQS queues.
  3. Different microservices process the queues.

Advantages:

  • Scalable architecture
  • Reliable message processing
  • Independent services

16. When to Use SQS

Use SQS when:

  • One service must process each message
  • Tasks should be processed asynchronously
  • Systems must be decoupled
  • Workloads need buffering

Example:

Background job processing systems.


17. When to Use SNS

Use SNS when:

  • Multiple systems must receive the same event
  • Real-time event notifications are needed
  • Applications follow an event-driven design

Example:

Event notification systems.


18. Benefits of Messaging Systems in AWS Architectures

Messaging systems help achieve scalable and loosely coupled architectures, which is a key concept in the SAA-C03 exam.

Key benefits:

Decoupling

Services can work independently.

Scalability

Consumers can scale horizontally.

Reliability

Messages are stored until processed.

Fault Tolerance

Failures in one component do not break the system.

Asynchronous Processing

Tasks can run in the background.


19. Exam Tips for SAA-C03

Important points frequently tested in the exam:

SQS

  • Standard vs FIFO queues
  • Visibility timeout
  • Dead-letter queues
  • Long polling
  • Message durability

SNS

  • Publish/subscribe model
  • SNS topics
  • Fan-out pattern
  • Integration with SQS and Lambda

Architecture Design

  • Use SQS to decouple services
  • Use SNS to broadcast events
  • Combine SNS + SQS for scalable systems

Summary

Queuing and messaging systems allow application components to communicate asynchronously. AWS provides Amazon SQS for queue-based messaging and Amazon SNS for publish/subscribe messaging. These services help build scalable, reliable, and loosely coupled architectures, which are essential for designing cloud applications and are commonly tested in the AWS Certified Solutions Architect – Associate (SAA-C03) exam.

Buy Me a Coffee