Monitoring celery applications can be really very challenging, one simple solution is to use Flower🌺. Its a real-time monitoring solution for celery. Other options include rabbit-mq dashboard, if we use rabbit-mq as a broker. However, flower is an absolutely beautiful and efficient monitoring solution as a starting point.

Flower provides a web-based interface that allows you to monitor the tasks running on your Celery worker. The interface is split into several sections that provide different types of information for instance workers, queues, tasks, stats, and much more.

We need to add flower as a dependency in our requirements.txt file and do a pip install -r requirements.txt

uvicorn==0.21.1
fastapi==0.95.0 
redis==4.5.4
celery==5.2.7
python-dotenv==1.0.0

#new
flower==1.2.0 

Once its done, the flower will take up the heavy responsibility of integrating with celery and communicating with it. All we need to do is to spin up a celery-flower server.

celery -A main.celery flower --port=5555

If everything worked fine, you should see a similar output:

[I 230415 01:17:19 command:162] Visit me at http://localhost:5555
[I 230415 01:17:19 command:170] Broker: redis://127.0.0.1:6379/0
[I 230415 01:17:19 command:171] Registered tasks: 
    ['02_celery_basics.main.send_push_notification',
     'celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap']
[W 230415 01:17:19 command:177] Running without authentication
[I 230415 01:17:19 mixins:225] Connected to redis://127.0.0.1:6379/0

Now, we can visit: http://localhost:5555/ and send a web request to fastapi server using docs/postman and see the progress of our tasks in realtime.

Don't forget to explore http://localhost:5555/tasks. Visit this tab and click on UUID of any of the available tasks to get detailed information of the task.