YT YouTube2Text

YouTube2Text → Transcripts → AWS SQS vs SNS Explained! | Application Integration (Day 15)

AWS SQS vs SNS Explained! | Application Integration (Day 15) — Transcript

by Pawan Joshi · 3,618 words · 654 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:02Hey everyone, welcome back and a huge
0:04congratulations because today we are
0:06kicking off week three of our journey.
0:08This is day 15 of 21, which means you
0:10are already past 2/3 of the way through
0:12this course.
0:13Give yourself credit for that.
0:15Today's topic is called application
0:17integration services, and honestly, this
0:19is one of my favorite topics in the
0:20entire course because it explains
0:23something that most beginners never
0:24really think about. How do different
0:26parts of an application actually talk to
0:28each other?
0:29When you open an app like Amazon or
0:30Swiggy or Zomato, you are not just
0:32talking to one giant program. Behind the
0:34scenes, there are dozens of small
0:35independent services. One service
0:37handles your login, another handles
0:39payment, another handles inventory,
0:41another sends you notifications.
0:44These services need a way to communicate
0:45and pass information to each other
0:47reliably, safely, and without one
0:48service crashing the whole system if it
0:50goes down.
0:51That's exactly what we are covering
0:53today. We will look at three core AWS
0:55services that let applications talk to
0:57each other.
0:58Amazon SQS,
1:00Amazon SNS,
1:02and Amazon EventBridge. Plus two more
1:05services that help you build and
1:06orchestrate modern applications. API
1:08Gateway
1:10and Step Functions.
1:12This falls under domain three, cloud
1:13technology and services, which is worth
1:16a massive 34% of your exam.
1:18The single biggest domain of
1:20this whole CLF-C02 exam. So, pay close
1:23attention because what you learn today
1:24is going to show up again and again in
1:26your exam questions. By the end of
1:28today, you will be able to clearly tell
1:29the difference between SQS, SNS, and
1:31EventBridge, and know exactly which one
1:33to pick in an exam scenario. You will be
1:35able to compare SQS standard queue
1:37versus FIFO queue, and know when to use
1:40each.
1:41You will understand what API Gateway and
1:43Step Functions actually do in a
1:45serverless architecture.
1:47And after the theory, we are jumping
1:48into a hands-on lab where you will
1:49create a real SQS queue in your own AWS
1:52account. You will send a message into it
1:54and receive it back. So, you are not
1:55just memorizing definitions. You're
1:57actually doing the thing.
1:59So, let's get into it.
2:01Before we jump into specific services, I
2:03want you to understand why this topic
2:05exist at all. Why does AWS even need
2:08queues and notification system? What is
2:10the problem are we actually solving
2:12here?
2:13Let's talk about tight coupling first.
2:15Imagine you build an e-commerce website
2:17the old-fashioned way. A single big
2:20application where the checkout process
2:22directly calls payment service
2:25which directly calls inventory service
2:30which directly calls the shipping
2:31service all in a long connected chain.
2:35This is called a tightly coupled
2:36architecture.
2:38Now, here's the problem. What happens if
2:39the shipping service goes down for
2:41maintenance at 2:00 a.m.?
2:43In a tightly coupled system, the
2:44complete the entire chain breaks.
2:47Your customer can't even complete a
2:49purchase even though shipping wasn't
2:50even needed yet at that stage. This is
2:53what we call a
2:54brittle chain.
2:56One single point of failure can bring
2:58down the whole system and failures
3:00cascade through the entire architecture
3:01like dominoes falling.
3:04You must have seen those.
3:05Now, compare that to a loosely coupled
3:07architecture. What do you mean by that?
3:09Here, each component doesn't talk
3:11directly to the next one. Instead, they
3:13communicate through a buffer layer. A
3:15middleman that holds messages safely
3:17until the next service is ready to
3:19process them. If the shipping service
3:21goes down for 10 minutes, no problem.
3:23The order just waits safely in a queue.
3:25And the moment shipping comes back
3:26online, it picks up right where it left
3:28off.
3:30Nothing is lost. Nothing crashes. The
3:31customer checkout is still complete
3:33successfully.
3:34This is the entire philosophy behind
3:36today's lesson. Buffered components are
3:38resilient, independently scalable, and
3:40fault tolerant.
3:42Each piece of your application can fail,
3:44restart, or scale up, scale down on its
3:46own without taking the rest of system
3:48down with it.
3:49Okay, so here's exactly what we are
3:51covering today step-by-step.
3:53Number one, the case of decoupling.
3:55Which we just covered why tight coupling
3:57is a reliability risk.
3:59Number two, SQS vs. SNS.
4:03This is without a doubt the single most
4:05important comparison in this entire
4:06domain for your exam. If you remember
4:08only one thing from today, remember this
4:10one.
4:12Number three, SQS standard vs. FIFO
4:14queues. Two flavors of the same service
4:16and knowing when to use which one is a
4:18very common exam trap.
4:20Number four, a quick but important
4:21overview of EventBridge, API Gateway,
4:24and Step Functions.
4:26Number five, our hands-on lab, where we
4:28will actually create an SQS queue, send
4:30a message into it, receive it, and
4:32delete it. The full life cycle.
4:34All right, let's start with a big one.
4:37All right, this slide has an exam focus
4:38tag on it for a reason. Questions
4:40comparing SQS and SNS show up constantly
4:43on the real exam. So, let's build a
4:44rock-solid mental picture of both.
4:47First, let's talk about Amazon SQS.
4:50Simple Queue Service.
4:52Think of SQS like a postal mailbox.
4:55Imagine you drop a letter into a
4:56mailbox, that letter sits there safely
4:58until exactly one person, the postman or
5:01whoever owns that mailbox, comes and
5:03picks it up.
5:04Once it is picked up and read, the
5:06letter is gone. Nobody else gets a copy
5:08of it.
5:10That's exactly how SQS behaves.
5:13It's called a point-to-point messaging
5:15pattern, and here's the flow
5:16step-by-step.
5:17A producer, some part of your
5:19application, places a message into the
5:21queue.
5:22The queue stores that message durably,
5:24meaning it's saved safely, even if
5:26nothing is reading it yet.
5:28A consumer, another part of your
5:29application, comes along, retrieves the
5:31message, and processes it.
5:34Once processing is done, the consumer
5:36deletes the message from the queue.
5:38The key idea to lock into your brain
5:40here is one message goes to one
5:42consumer, and it's a consumed exactly
5:44once. Even if you have multiple
5:46consumers attached to the same queue,
5:48each individual message will only ever
5:50be picked up and processed by only one
5:52of them.
5:53Not all of them.
5:55Let's take a real use case.
5:56Imagine an e-commerce site. A customer
5:59places an order.
6:00That order placed event goes into an SQS
6:03queue.
6:04On the other side, your payment
6:05processing service picks up orders from
6:07that queue one at a time in a controlled
6:09sequential way, processing them without
6:11getting overwhelmed by the sudden burst
6:13of traffic.
6:14SQS acts like a shock absorber. Even if
6:1610,000 orders come in during a flash
6:18sale, they queue up nicely and get
6:20processed steadily instead of crashing
6:22your payment system.
6:24Now, let's talk about Amazon SNS, that
6:26is Simple Notification Service.
6:29If SQS is a postal mailbox, think of SNS
6:32like a broadcast announcement.
6:34Like a school PA system or a group email
6:37blast.
6:38When one message goes out, everybody who
6:40is subscribed hears it at the same time.
6:42SNS works on something called a pub/sub
6:45pattern, which stands for
6:46publish/subscribe.
6:49There's a publisher
6:51who sends out a single message to
6:53something called an SNS topic.
6:57That topic can have multiple subscribers
6:59attached to it, and when the message is
7:01published, every single subscriber gets
7:03a copy of it simultaneously.
7:05This is also called a fan out pattern
7:08because one message fans out to many
7:09destinations at once.
7:11Let's take one real use case. Let's say
7:13a new user signs up on the application.
7:16That user registered event gets
7:18published to an SNS topic.
7:20Now, at the exact same moment, an email
7:22service subscriber sends the user a
7:23welcome email.
7:25An analytics service subscriber logs
7:27this new registration event for
7:28reporting.
7:29A CRM system subscriber updates the
7:31customer relationship database with this
7:33new user details.
7:35All three of these things happen
7:37simultaneously, triggered by that one
7:39single publish action. None of them wait
7:41for each other.
7:42That's the power of fan out.
7:44So, here's the one-line summary you
7:45should memorize for the exam.
7:47SQS is equal to
7:50one message, one consumer,
7:53point-to-point.
7:55Message deleted after
7:57being processed.
8:00SQS equal to
8:02one message,
8:04many subscribers,
8:07all triggered at the same time, fan out
8:10pattern.
8:11If an exam question says something like,
8:13"We need to decouple our order
8:14processing so a single service can pick
8:16up and process each order."
8:18That's SQS. But, if the question says,
8:21"We need to notify multiple different
8:23systems at once when an event happens."
8:26That's SNS.
8:29Now that you have understood what SQS
8:31is, I need to tell you something
8:33important.
8:34SQS actually comes in two different
8:36flavors, and choosing the wrong one in
8:38the real system or on your exam can
8:40completely outcome. Let's break them
8:43down.
8:44We have a standard queue. This is the
8:46default type of SQS queue, and it's
8:48built for raw speed and massive scale.
8:51It has three defining characteristics.
8:54At least once delivery. This means a
8:57message is guaranteed to be delivered,
8:59but in rare cases it might accidentally
9:01be delivered more than once. Yes,
9:03duplicates are technically possible.
9:06The another one is best effort ordering.
9:09AWS tries its best to keep messages in
9:11the order you sent them, but it's not
9:13guaranteed. Messages might occasionally
9:15arrive out of sequence.
9:18Virtually unlimited throughput. Standard
9:20queues can handle an enormous, almost
9:22unlimited number of transactions per
9:24second.
9:26Where would you use a standard queue?
9:28Anywhere high volume matters more than
9:30perfect precision. Think log processing,
9:32best data uploads, or any fault tolerant
9:34workload where occasionally processing
9:36the same thing twice isn't a big deal.
9:39Now, let's discuss FIFO queue.
9:41FIFO stands for first-in, first-out,
9:44meaning messages come out in exactly the
9:45same order they went in. No exceptions.
9:48This queue type is built for precision,
9:50not raw speed. Its characteristics are
9:53exactly once processing, no duplicates
9:55guaranteed. A message will be processed
9:58one time and only one time.
10:01Strict message ordering. Messages are
10:03always delivered in the exact sequence
10:05they were sent.
10:07Limited throughput, a maximum of 3,000
10:09transactions per second when using
10:11batching. That's still fast, but nowhere
10:13nearly the virtual unlimited throughput
10:15of a standard queue.
10:16Here's something AWS strictly enforces.
10:19The name of every FIFO queue must end
10:20with the .fifo suffix.
10:23If you try to create a FIFO queue
10:24without that suffix, AWS will simply
10:26reject it. This is a small detail, but
10:28exams love testing small enforced
10:30details like this. So, where would you
10:32use a FIFO queue?
10:34Anywhere that duplicate processing would
10:36be a disaster. Think about financial
10:37transactions. If a customer credit cards
10:40get accidentally charged twice because
10:41of a duplicate message, that's a serious
10:44problem
10:45both for the customer and for your
10:46business.
10:47Same with e-commerce order processing.
10:49You never want that the same order
10:51accidentally submitted twice.
10:53Here's the exam pattern you must
10:54memorize. Whenever you see a question
10:56that mentions no duplicate processing,
10:59exactly once, or financial transactions
11:02that must not be processed twice,
11:05the answer is almost always SQS FIFO
11:07queue.
11:08Keep that phrase logged in your memory.
11:09No duplicates, financial transactions
11:11equals FIFO. And don't forget that the
11:14queue name has to end with .fifo suffix.
11:18Now, let's quickly cover three more
11:20services that round out the application
11:22integration picture.
11:23These come up less frequently than SQS
11:25versus SNS on the exam, but you
11:27absolutely need to know what each one
11:29does and, more importantly, what makes
11:31each one different from the others?
11:34Amazon EventBridge. Think of EventBridge
11:36as a serverless event bus. Basically, a
11:39smart traffic controller for events
11:41happening across your AWS environment.
11:43You create rules that say, "When this
11:45specific event occurs, automatically
11:47route it to that specific target."
11:50Let's take a real example.
11:52Say an EC2 instance gets terminated.
11:55You can set up an EventBridge rule that
11:56says, "Whenever an EC2 instance
11:58terminates, automatically trigger a
12:00Lambda cleanup function." Maybe to clean
12:02up leftover resources, update a
12:03database, or send an alert.
12:06This all happens automatically without a
12:08human lifting a finger.
12:10EventBridge isn't just for AWS own
12:12internal events, either. It can also
12:13react to events from SaaS applications
12:15like you must have heard about Zendesk
12:17or Shopify, and even your own custom
12:20application events.
12:21Basically, if something happens,
12:22EventBridge can catch it and route it
12:24wherever it needs to go
12:26based on the rules you define in it.
12:29The next one is Amazon API Gateway.
12:32Think of API Gateway as the front door
12:33of your backend services.
12:35When outside applications, mobile apps,
12:37or website wants to talk to a backend,
12:39they don't talk to it directly. They go
12:42through an API Gateway first.
12:44API Gateway lets you build and manage
12:46REST APIs, HTTP APIs, and WebSocket
12:49APIs. And it comes packed with useful
12:52built-in features so you don't have to
12:53build them yourself.
12:55It has built-in rate limiting,
12:57authorization, catching, request
12:59validation, and SSL termination. What do
13:01these mean? Rate limiting means
13:02controlling how many requests a client
13:04can make to prevent abuse or overload.
13:07Authorization means checking whether the
13:09person calling your API is actually
13:11allowed it to.
13:13Catching
13:15is storing frequent responses so your
13:17backend doesn't get hit with repeat
13:18requests.
13:20And we have request validation. That
13:21means making sure incoming data is
13:23properly formatted before it even
13:25reaches your backend.
13:27And talking about SSL termination, it
13:28means handling the secure HTTPS
13:30encryption or decryption, so your
13:32back-end servers don't have to.
13:34Basically, API Gateway sits at the entry
13:36point of your architecture and handles
13:38all the traffic management work before
13:40request even reach your actual
13:41application logic.
13:43The next service we have is AWS Step
13:45Functions.
13:46Step Functions is all about workflow
13:48orchestration using something called a
13:49state machine.
13:51Imagine you have a complex process with
13:53many steps. Maybe a validate an order,
13:56check inventory, charge the payment,
13:58update the database, then send a
13:59confirmation email as well.
14:01Instead of writing messy code to
14:03manually chain all these steps together,
14:05Step Functions lets you visually design
14:07the entire workflow step-by-step. And it
14:10comes with powerful building
14:11capabilities.
14:13We have retry logic, that means
14:14automatically retrying a step if it
14:16fails, instead of the whole workflow
14:18crashing.
14:19We have error handling, that is
14:21gracefully catching failures and
14:22deciding what to do next.
14:24Conditional branching,
14:26that is taking different path depending
14:28on the outcome of a step. Like an if
14:31this, then that logic.
14:33Another is parallel execution, running
14:35multiple steps at the same time when
14:37they don't depend on each other.
14:39And here's a very common real-world
14:40combo you should remember.
14:42SQS plus Lambda is probably the most
14:44common serverless message processing
14:46pattern you will see in real AWS
14:47architectures. Here's how it works.
14:50Messages land in an SQS queue, and the
14:52AWS Lambda automatically pulls that
14:54queue in the background, picking up
14:56messages as they arrive and processing
14:57them. Completely serverless with no
15:00server management required on your end.
15:03This combination shows up constantly in
15:04real production systems, and it's a
15:06pattern worth remembering both for your
15:07exam and for your future projects as
15:09well.
15:14All right, theory is done. Now, let's
15:15get our hands dirty. This is where
15:18things get really fun because you are
15:19going to to
15:20and use a real SQS queue with your own
15:22hands right inside your AWS account.
15:25Make sure you're logged in into AWS
15:26Management Console before we start and
15:28let's go step-by-step together.
15:31First in the AWS Console search bar at
15:33the top, type in SQS
15:37and click on Amazon SQS to open the
15:39service.
15:41And once you're in the SQS dashboard,
15:43click on create queue button.
15:46Now you will see two options, standard
15:48or FIFO.
15:50For this lab we are going to select
15:52standard since we just want to
15:54understand the basic message life cycle
15:56first.
15:57And for the name, type in
15:59my
16:01test queue.
16:03You will notice a setting called
16:04visibility timeout. Leave this at the
16:07default of 30 seconds. I want you to
16:09remember this setting because it's
16:10actually really important and I will
16:12explain exactly why in a moment.
16:15Uh leave every other setting all at its
16:17default value and click on create queue
16:19at the bottom.
16:21All right, congratulations. You just
16:23created your very first SQS queue.
16:27Now click on the queue that you just
16:29created to open it and then click on the
16:31button that says send and receive
16:33messages.
16:34And in the message body, type
16:37uh this like hello from
16:40SQS.
16:42All right, now click on send message.
16:45You should see a success notification
16:47pop up along with this uh unique message
16:49ID.
16:50This confirms that your message was
16:51successfully stored in the queue.
16:53And if you scroll down and check uh this
16:56metric,
16:57uh you can see the receive message
16:59section, there is uh it is showing one
17:00message and confirming there is exactly
17:03one message sitting in the your queue
17:04right now waiting to be picked up.
17:06So in the receive message section only,
17:09uh click on the button labeled poll for
17:11messages.
17:13This is basically you acting as a
17:15consumer and you are asking the queue,
17:16"Hey, do you have anything for me?" And
17:18within a moment, your message would
17:19appear in the list below.
17:21Here, click on the message.
17:23Now, you can see you will be able to
17:25read the full message body that you
17:26typed exactly earlier.
17:29Now, click on delete. Let's delete it.
17:33Uh
17:33So, it will remove the message from the
17:35queue permanently.
17:37So, here's the important part I promised
17:38to explain,
17:40the visibility timeout. When you poll
17:42for a message and it appeared to you,
17:44SQS didn't just leave it to uh visible
17:46to everyone. It temporarily hid that
17:47message from any other consumer for 30
17:49seconds, the visibility timeout we set
17:51earlier.
17:52This exists specifically to prevent
17:54duplicate processing. Imagine if two
17:57different consumers grab the exact same
17:59message at the same time and both try to
18:01process it.
18:02That could cause serious problems, like
18:04charging a customer twice.
18:06So, here's a golden rule. If you don't
18:08delete the message within the uh that
18:1030-second visibility timeout window, it
18:12automatically reappears in the queue,
18:14becoming available for another consumer
18:16to pick up. This is by design. It's
18:19SQS's way of guaranteeing that messages
18:21never get silently lost, even if the
18:23consumer get that grabbed them crashes
18:25or fails before finishing the job.
18:28Okay, now if you want to go a step
18:30further, let's click on create queue
18:32again.
18:33But this time, select FIFO.
18:36For the name, let's type my FIFO
18:40. FIFO.
18:42Notice something here? Uh AWS will not
18:45let you create unless the name ends with
18:47.FIFO.
18:49This is exactly the enforced naming rule
18:51we talked about earlier in the theory
18:53section.
18:54Try removing that suffix and you will
18:55see AWS immediately throws an error.
18:58While setting this up, uh you can also
19:00enable a feature called content-based
19:02deduplication,
19:04which tells AWS to automatically detect
19:06and eliminate duplicate messages based
19:08on their content. Handy feature that
19:10enforces the exactly one guarantee of
19:12FIFO queues.
19:13Go ahead and click create queue and then
19:16try sending a few messages into this
19:18FIFO queue one after another.
19:20When you receive them, notice how they
19:21always come out in the exact same order
19:24you sent them. That strict ordering
19:26guarantee in action, something a
19:27standard queue can't promise you.
19:29All right, so take your time with this
19:31lab, play around with it, send a few
19:33more messages, delete a few, and get
19:35really comfortable with how a queue
19:37behaves.
19:38Because this hands-on understanding is
19:40going to help you far more than just
19:41memorizing the definitions.
19:44First, SQS versus SNS.
19:47SQS is a message queue that works on a
19:49point-to-point pattern. Each message is
19:51consumed exactly once by one consumer.
19:55And SNS is a pub/sub notification system
19:58that works on a fan-out pattern. That
20:00is, one message goes out to all
20:02subscribers simultaneously.
20:04Second, standard versus FIFO queues.
20:07A standard queue gives you virtually
20:09unlimited throughput, but duplicates are
20:11possible and ordering isn't guaranteed
20:12as well.
20:14A FIFO queue guarantees exactly-once
20:17processing and strict ordering.
20:19But at a lower throughput cap of 3,000
20:22transactions per second. And its name
20:24must always end in .fifo.
20:28Third, the wider integration ecosystem.
20:30EventBridge handles event-driven
20:32routing, reacting automatically to
20:34events and routing them to targets.
20:36API Gateway acts as a front door API
20:39layer for your back-end services. And
20:41Step Functions handles multi-step
20:43workflow orchestration, coordinating
20:44complex processes with retry logic and
20:46error handling built right in.
20:49So, for tonight practice, go and attempt
20:5110 questions which are in the notes and
20:54pay close attention to how questions are
20:56worded. Like,
20:57words like decouple, notify multiple
21:00systems, no duplicates, and event-driven
21:02are all clues pointing toward a specific
21:05service.
21:06The exam loves testing these subtle
21:08keyword patterns and so the more you
21:10practice spotting them, the faster and
21:12more confidently you will answer the on
21:13the exam day.
21:15And a quick look ahead, tomorrow is day
21:1716 and we are moving into security
21:19tools. We will be covering guard duty,
21:21WAF, shield, Macy, inspector, KMS and
21:25secret manager.
21:27Yes, that's a lot of services in one
21:28day, but don't worry, each one maps very
21:30clearly to a specific security use case.
21:33And once you see the pattern, it all
21:35clicks into place quickly.
21:37That's it for day 15. You now understand
21:39how modern resilient AWS applications
21:41talk to each other behind the scenes.
21:44And you have built a working SQS queue
21:46with your own hands. That's real
21:48practical progress. Great job today. Go
21:50finish up those practice questions, get
21:52some rest and I will see you tomorrow
21:53for day 16. Keep going, keep doing. You
21:56are doing great.

About this transcript

This page contains the full transcript of AWS SQS vs SNS Explained! | Application Integration (Day 15) by Pawan Joshi, generated from the public captions YouTube serves with the video. The transcript has 3,618 words across 654 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.