Auto-reloading celery when files are modified

When working on a Celery project, We may find ourselves making frequent changes to the code. This can be a problem because Celery does not automatically reload the tasks when you make changes to the code. This can be frustrating at times, as when working with fastapi, the uvicorn server reloads automatically and picks up the changes but celery won't 😑. In this post, we will discuss how to auto-reload Celery when files are modified.

All we need to do is to use some utility tool that monitors file system events and restarts celery on file change. Feel free to use any tool of your choice e.g. watchdog, watchfiles, etc. In this one, we are going to use watchdog 🐕. Let's modify our requirements.txt file to include it and do a pip install -r requirements.txt

uvicorn==0.21.1
fastapi==0.95.0 
# ...

#new
watchdog==3.0.0

Once it's successfully installed, we can use the same celery command on top of the watchdog to monitor.

watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery -A main.celery worker --loglevel=info

Basically, we are telling watchdog to keep an eye on any file that ends with .py, Once a file is modified, kill the celery process and run the celery command again.