Why Task Queues ?

Why?
Why do we even need task queue? Yes I am not starting with What is task queue? because why will answer it better!

Let's imagine we are running a payment gateway company like Stripe/Paypal. We have a big client which has millions of transactions in their acount. Lets say someday this client wants to download their last month's transaction report. They click on a button which redirects them to a page with a beautiful and creative loading icon!

Wow its creative and beautiful. Its been around 12 seconds and the above loading icon is still there. Now the clients starts panicing a bit.
After few seconds they see a 404 message on the screen! What just happened? Downloading the reports had several steps involved:

Extracting Transactions data from Database
Writing data to an excel file.
Uploading the excel file to Blob store e.g. S3
Passing the URL link to frontend

Each of these are extremely time consuming steps because they require I/O. Talking to databse is around thousand times slower than talking to RAM, Making a network call to upload a file are generally even slower than that. Thus, it is very natural that our end users will have to wait a lot and in worst cases they will even get a 404 error as the server could not respond back to the request. This is one of the use cases, which is best handled by task queues.

Other general use cases:

Sending Emails: Sending emails requires talking to a SMTP server, which is time consuming. If we have to send multiple emails, it can be aweful experience to our website users.
Generating Reports: PDF/Excel Report generation takes several seconds.
Uploading Image: Uploading image to a blob store e.g. S3.
Network Calls: Sending a web request can be expensive and may have high failure rate. Sometimes we might need to even retry the request.
Anything that does not need to be done instantly but can be done eventually!

Back to the our intial problem, This time the client clicks on the button to download monthly report. They see a message on the screen saying "We are preparing the report, You will receive it on [email protected] (Task ID:12345)". After 3-4 mins they receive the email.

This is an example of how task queues can help us. To know exactly how you can do this, come abord on this wonderful jouney of understanding task queues.
Happy Journey 🛥️