Deploying Celery workers with Supervisor

Though, in the previous tutorial, we were able to gracefully serve our API. If you visit http://your-ip/push/123 and keep your logs open you might see a similar 500 Internal server error.

johnwick@fastapixcelery:~$ cd project/logs/
johnwick@fastapixcelery:~/project/logs$ tail -f fastapi_gunicorn.log 

  File "/home/johnwick/project/env/lib/python3.10/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
    raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: Error 111 connecting to 127.0.0.1:6379. Connection refused.

It says that our redis/broker is not reachable at port 6379. Indeed that is the case, we haven't done anything to configure celery. Let's start with installing redis on the remote server.

johnwick@fastapixcelery:~/project/logs$ sudo apt install redis-server
johnwick@fastapixcelery:~/project/logs$ sudo systemctl restart redis.service
johnwick@fastapixcelery:~/project/logs$ redis-cli
127.0.0.1:6379> ping
PONG

Note: In case you face an error in restarting Redis service, You might need to open /etc./redis/redis.conf and change the line that says supervised no to supervised systemd.
Use sudo to open up the file: sudo nano /etc/redis/redis.conf

Otherwise, we are good to go. The way we created gunicorn_conf.bash file and hooked it to the supervisor, In a similar way, we need to create celery conf and hook it to the supervisor.

johnwick@fastapixcelery:~$ cd  project/infra/
johnwick@fastapixcelery:~/project/infra$ ls
gunicorn_conf.bash  run
johnwick@fastapixcelery:~/project/infra$ touch celery_high_priority_worker.bash
johnwick@fastapixcelery:~/project/infra$ chmod +x celery_high_priority_worker.bash

johnwick@fastapixcelery:~/project/infra$ nano celery_high_priority_worker.bash
#paste the below celery configuration in this file

We need to put the configuration in this the above bash file. This will define how to start a celery worker corresponding to high_priority queue. If you don't have configured queues in your codebase, use the default queue name i.e. "celery" instead of high_priority.

#!/bin/bash

NAME="main.celery"
NODE_NAME="highpriorityqueue"
echo "Starting $NAME as `whoami`"


DIR=/home/johnwick/project
VENV=$DIR/env/bin/activate

cd $DIR
source $VENV

exec celery  -A $NAME worker -l info -n $NODE_NAME  -Q high_priority

Superrrr, now our configuration file is done all we need now is to hook it to the supervisor.

johnwick@fastapixcelery:~/project/infra$ cd /etc/supervisor/conf.d/

johnwick@fastapixcelery:/etc/supervisor/conf.d$ ls
gunicorn.conf

johnwick@fastapixcelery:/etc/supervisor/conf.d$ sudo touch celery_high_priority.conf

johnwick@fastapixcelery:/etc/supervisor/conf.d$ sudo nano celery_high_priority.conf 

Put the below configuration in the celery_high_priority.conf file:

[program:celery_high_priority]
command=/home/johnwick/project/infra/celery_high_priority_worker.bash
user=johnwick
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/home/johnwick/project/logs/celery_high_priority.log

Once, all this configuration is set up, we need to tell the supervisor to update and read the new configuration. To be safe, let's restart and check the status as well. Now, we are good to visit /push/123 and we should see the task logs in celery_high_priority.log file.

That's what is needed. Yes, we did not set up low_priority_worker, periodic worker, and celery beat configuration. I would suggest giving it a try on your own. Since the overall format is exact the same. i would love to suggest it as an assignment.
 

Assignment:

• create a low priority worker
• create a periodic worker
• create a supervisor conf for starting beat server
• create a supervisor conf for managing low_priority worker bash file as command.