Retrying Celery Tasks 🌋

Things often does not work as planned. We need to retry!
- Celery Worker

While Celery is generally reliable, there may be times when a task fails due to network issues, unavailability of resources, or other unforeseeable issues. Let's take an example, A new user signs up at our website and we need to push their detail in external services e.g. Mailchimp,Zoho etc. So, as to retarget for marketing.  Many times, these external API calls can fail. It could be a downtime or error in external API or there can be n number of reasons.

In my last chatbot company, I had to build an event-based webhook system that sends the user's information to our client's API as soon as users enter their phone number in chat. If the client's API is down, the request would fail. To handle it, We have to design a robust background task mechanism by using retrying...

Let's modify our task in tasks.py  to:
 

import time
import random
from celery import shared_task
from celery.utils.log import get_task_logger

logger = get_task_logger("tasks")


@shared_task(bind=True)
def send_notification(self,device_token: str):
    try:
        logger.info("starting background task")
        time.sleep(2)  # simulates slow network call
        if random.choice([0,1]):
            raise Exception()
    except Exception as e:
        raise self.retry(exc=e, countdown=10, max_retries=3)

We basically passed a new argument called bind. The benefit is that, now, we have access to the task type instance self.
In the above code are randomly raising exceptions, so if you trigger this task, sometimes, it will succeed and sometimes it would fail.
In case the task fails, it would raise an exception, wait for 10 seconds to do a retry, and will retry a maximum of 3 times.