Debugging Celery Task : RDB

Every used pdb or ipdb? These are interactive source code debuggers for Python. They stop the flow of code on the specified breakpoint and then we can use some of the utilities for efficient debugging.

• n to move to the next line.
• s to step inside a function
• c to continue to end/move to the next breakpoint.
• q to exit

def send_notification(device_token: str):
    timeout = cache.get(f"timeout_{device_token})
    
    import pdb;
    pdb.set_trace() # stops the flow at terminal for debugging
    
    if timeout:
        return "Token is expired"

    if not timeout:
        send_msg_by_firebase()

    logger.info(f"notification sent {device_token}")

Pdb is a super essential utility, especially in a codebase that is newer to us or the codebase is messy. In such times, I personally used pdb to go through line by line and understand the code and fix a bug.

The only issue is that we will not be able to use pdb/ipdb in our celery tasks. Again the same reason, these tasks are not being handled in the same Python thread and the context of callbacks are missing. We can utilize another celery utility named rdb💣. It stands for remote debugging bomb!
Joking, it's remote debugger.

Let's try it out, modify the background task as below:

from celery.contrib import rdb

def send_msg_by_fcm(device_token):
    logger.info(f"sent using fcm {device_token}")
    return True


@celery.task
def send_notification(device_token: str):
    logger.info("starting background task")
    time.sleep(5)  # simulates slow network call
    rdb.set_trace()
    if device_token:
        sent = send_msg_by_fcm(device_token)
    a=11/0 
    logger.info(f"notification sent {device_token}")

Start the uvicorn and celery instance with celery -A main.celery worker --loglevel=info

Make a request to the endpoint that calls send_notification with .delay().
If you check the celery terminal, you should see the processing is stopped and there should be the below message.

Task main.send_notification[00872aed-e082-4a16-9464-5a252d5e1776] received

Remote Debugger:6907: Ready to connect: telnet 127.0.0.1 6907

Type `exit` in session to continue.

Remote Debugger:6907: Waiting for client...

Copy the telnet 127.0.0.1 6907 and paste it in another terminal. You should be in the rdb debugger now, feel free to type the name of variables and check their values, enter n to move to the next line, and c to complete execution and close it.