『SQS Dead Letter Queue Storm: Diagnose and Recover』のカバーアート

SQS Dead Letter Queue Storm: Diagnose and Recover

SQS Dead Letter Queue Storm: Diagnose and Recover

無料で聴く

ポッドキャストの詳細を見る
概要
 You wake up to 10,000 messages in your SQS dead letter queue — here's the systematic CloudWatch-driven approach to diagnosing the root cause and safely replaying messages without triggering a second storm.You'll learn:Which four CloudWatch metrics to pull first (ApproximateNumberOfMessagesVisible, NumberOfMessagesReceived, NumberOfMessagesDeleted, ApproximateAgeOfOldestMessage) and how to read the curve shape to distinguish a deploy failure from resource exhaustionWhy redriving immediately is the most common wrong answer — and how to confirm the root cause is actually resolved before touching the DLQHow visibility timeout math interacts with Lambda timeouts to silently manufacture poison pill storms at scaleUsing StartMessageMoveTask (or the console's Start DLQ redrive) for rate-controlled replay instead of a manual receive-delete-republish loopHow to set maxReceiveCount deliberately — why 1-2 is too aggressive and 10 is too lax, and how to align visibility timeout to your p99 processing timeKeywords: SQS dead letter queue, DLQ redrive policy, SQS visibility timeout, AWS SQS CloudWatch metrics, SRE interview questions AWS🎧 Listen, then go deeper — DevOps & Cloud interview-prep ebooks at DevOpsInterview.Cloud▶ Daily 30-second interview drills: DevOps Interview Cloud on YouTubeTranscriptYou wake up to a Slack alert. Your dead letter queue, the DLQ, has ten thousand messages sitting in it. Somewhere overnight, a queue that normally holds zero items quietly filled up, and nobody got paged until the number crossed a threshold. This is one of those incidents that looks small until you realize it might mean ten thousand failed payments, ten thousand missed notifications, or ten thousand orders stuck in limbo.Interviewers love this scenario because it tests three things at once. Can you read CloudWatch metrics fast enough to find the actual root cause instead of guessing. Do you understand how a redrive policy and visibility timeout interact. And can you recover the messages without making the outage worse by replaying them straight into the same failure loop.Start with the mental model. In SQS, every queue can have a redrive policy pointing to a dead letter queue, along with a maxReceiveCount. Every time a consumer pulls a message and fails to delete it, either by crashing, throwing an error, or just running past the visibility timeout, the receive count on that message goes up by one. Once it crosses maxReceiveCount, SQS automatically moves that message to the DLQ. The DLQ is not where messages go when something is broken right now, it is where they go after they have already failed a set number of times. So by the time you see ten thousand messages there, the failures happened earlier and repeated, meaning you are looking at an aftermath, not a live fire.First move: resist the instinct to immediately redrive the messages back to the source queue. That is the single most common wrong answer, and it will just refill the DLQ again if the underlying cause is still active. Diagnose first.Open CloudWatch and pull four metrics for the source queue: ApproximateNumberOfMessagesVisible, NumberOfMessagesReceived, NumberOfMessagesDeleted, and ApproximateAgeOfOldestMessage. Then pull ApproximateNumberOfMessagesVisible for the DLQ itself, graphed over the last twelve hours. You are looking for the shape of the curve. A sudden vertical jump around a specific timestamp tells you something changed at that moment, like a deploy, a dependency outage, or a config push. A slow staircase climbing over hours usually points to a resource exhaustion problem, like a database connection pool filling up under load.Next, check your consumer's own logs or traces around that timestamp. In a Lambda-based consumer, look at Errors and Throttles in CloudWatch for the function, plus Duration against your function timeout. A classic failure pattern: someone set the Lambda timeout to three seconds, a downstream API started taking four seconds under load, every invocation started timing out, SQS saw the message never get deleted, the visibility timeout expired, the message came back, failed again, and after three or five retries it landed in the DLQ. Multiply that by every message hitting the same slow endpoint and you get your ten thousand.Another classic: a single malformed message, a poison pill, causes a parsing exception on every attempt. But a poison pill alone usually only produces one or two hundred failures before someone notices, not ten thousand, unless the queue has heavy throughput. If you see ten thousand nearly identical failures with the same exception type and the same payload shape, that is a code or schema problem, not a flaky dependency. If you see varied payloads but a shared downstream timeout pattern, that is capacity, not code.Once you know the category, check whether it is still happening. Look at the error rate for the last fifteen minutes, not twelve hours ago. If errors are still climbing, fix the root ...
adbl_web_anon_alc_button_suppression_t1
まだレビューはありません