01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

Asynchronous Tasks With Django and

Celery

How to integrate
Celery into a Django Project and create Periodic Tasks.

Overview

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 1/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

Step Overview Git Tag

What is Celery?

Why is this useful?

Setup

1 $pipinstallcelery==3.1.18
2 $pipfreeze>requirements.txt

Step 1: Add

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 2/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

1 from__future__importabsolute_import
2 importos
3 fromceleryimportCelery
4 fromdjango.confimportsettings
5
6 #setthedefaultDjangosettingsmoduleforthe'celery'program.
7 os.environ.setdefault('DJANGO_SETTINGS_MODULE','picha.settings')
8 app=Celery('picha')
9
10 #Usingastringheremeanstheworkerwillnothaveto
11 #pickletheobjectwhenusingWindows.
12 app.config_from_object('django.conf:settings')
13 app.autodiscover_tasks(lambda:settings.INSTALLED_APPS)
14
15
16 @app.task(bind=True)
17 defdebug_task(self):
18 print('Request:{0!r}'.format(self.request))

Step 2: Import your new Celery app

1 from__future__importabsolute_import
2
3 #Thiswillmakesuretheappisalwaysimportedwhen
4 #Djangostartssothatshared_taskwillusethisapp.
5 from.celeryimportappascelery_app

1 manage.py
2 picha
3 __init__.py
4 celery.py
5 settings.py
6 urls.py
7 wsgi.py
8 requirements.txt

Step 3: Install Redis as a Celery Broker

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 3/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

brewinstallredis

1 $redisserver

1 $rediscliping

PONG

1 #CELERYSTUFF
2 BROKER_URL='redis://localhost:6379'
3 CELERY_RESULT_BACKEND='redis://localhost:6379'
4 CELERY_ACCEPT_CONTENT=['application/json']
5 CELERY_TASK_SERIALIZER='json'
6 CELERY_RESULT_SERIALIZER='json'
7 CELERY_TIMEZONE='Africa/Nairobi'

1 $pipinstallredis==2.10.3
2 $pipfreeze>requirements.txt

$celeryApichaworkerlinfo
1
...
2
[2015070714:07:07,398:INFO/MainProcess]Connectedtoredis://localhost:
3
6379//
4
[2015070714:07:07,410:INFO/MainProcess]mingle:searchingforneighbors
5
[2015070714:07:08,419:INFO/MainProcess]mingle:allalone

1 $celeryApichabeatlinfo
2 ...
3 [2015070714:08:23,054:INFO/MainProcess]beat:Starting...

Celery Tasks
https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 4/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

Celery Tasks

1 defadd(x,y):
2 returnx+y

1 fromcelery.decoratorsimporttask
2
3 @task(name="sum_two_numbers")
4 defadd(x,y):
5 returnx+y

1 add.delay(7,8)

feedback

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 5/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

1 feedback
2 __init__.py
3 admin.py
4 emails.py
5 forms.py
6 models.py
7 tests.py
8 views.py
9 manage.py
10 picha
11 __init__.py
12 celery.py
13 settings.py
14 urls.py
15 wsgi.py
16 requirements.txt
17 templates
18 base.html
19 feedback
20 contact.html
21 email
22 feedback_email_body.txt
23 feedback_email_subject.txt

Add the Task

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 6/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

1 fromcelery.decoratorsimporttask
2 fromcelery.utils.logimportget_task_logger
3
4 fromfeedback.emailsimportsend_feedback_email
5
6 logger=get_task_logger(__name__)
7
8
9 @task(name="send_feedback_email_task")
10 defsend_feedback_email_task(email,message):
11 """sendsanemailwhenfeedbackformisfilledsuccessfully"""
12 logger.info("Sentfeedbackemail")
13 returnsend_feedback_email(email,message)

1 fromdjangoimportforms
2 fromfeedback.tasksimportsend_feedback_email_task
3
4
5 classFeedbackForm(forms.Form):
6 email=forms.EmailField(label="EmailAddress")
7 message=forms.CharField(
8 label="Message",widget=forms.Textarea(attrs={'rows':5}))
9 honeypot=forms.CharField(widget=forms.HiddenInput(),required=False)
10
11 defsend_email(self):
12 #trytotrickspammersbycheckingwhetherthehoneypotfieldis
13 #filledinnotsupercomplicated/effectivebutitworks
14 ifself.cleaned_data['honeypot']:
15 returnFalse
16 send_feedback_email_task.delay(
17 self.cleaned_data['email'],self.cleaned_data['message'])

send_feedback_email_task.delay(email,message)

NOTE success_url /

Periodic Tasks

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 7/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

fromcelery.task.schedulesimportcrontab
1
fromcelery.decoratorsimportperiodic_task
2
3
4
@periodic_task(run_every=(crontab(minute='*/15')),name="some_task",ignore
5
_result=True)
6
defsome_task():
7
#dosomething

photos

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 8/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

1 feedback
2 __init__.py
3 admin.py
4 emails.py
5 forms.py
6 models.py
7 tasks.py
8 tests.py
9 views.py
10 manage.py
11 photos
12 __init__.py
13 admin.py
14 models.py
15 settings.py
16 tests.py
17 utils.py
18 views.py
19 picha
20 __init__.py
21 celery.py
22 settings.py
23 urls.py
24 wsgi.py
25 requirements.txt
26 templates
27 base.html
28 feedback
29 contact.html
30 email
31 feedback_email_body.txt
32 feedback_email_subject.txt
33 photos
34 photo_list.html

Add the Task

photos

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 9/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

1 fromcelery.task.schedulesimportcrontab
2 fromcelery.decoratorsimportperiodic_task
3 fromcelery.utils.logimportget_task_logger
4
5 fromphotos.utilsimportsave_latest_flickr_image
6
7 logger=get_task_logger(__name__)
8
9
10 @periodic_task(
11 run_every=(crontab(minute='*/15')),
12 name="task_save_latest_flickr_image",
13 ignore_result=True
14 )
15 deftask_save_latest_flickr_image():
16 """
17 SaveslatestimagefromFlickr
18 """
19 save_latest_flickr_image()
20 logger.info("SavedimagefromFlickr")

save_latest_flickr_image()
task @periodic_task

Running Locally

1 $celeryApichaworkerlinfo
2 $celeryApichabeatlinfo

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 10/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

photos/tasks.py

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 11/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

feedback/tasks.py

Running Remotely

1 $sudoaptgetinstallsupervisor

Celery Worker: picha_celery.conf

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 12/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

1 ==================================
2 celeryworkersupervisorexample
3 ==================================
4
5 thenameofyoursupervisordprogram
6 [program:pichacelery]
7
8 Setfullpathtoceleryprogramifusingvirtualenv
9 command=/home/mosh/.virtualenvs/picha/bin/celeryworkerApichalogleve
10 l=INFO
11
12 ThedirectorytoyourDjangoproject
13 directory=/home/mosh/sites/picha
14
15 Ifsupervisordisrunastherootuser,switchuserstothisUNIXusera
16 ccount
17 beforedoinganyprocessing.
18 user=mosh
19
20 Supervisorwillstartasmanyinstancesofthisprogramasnamedbynump
21 rocs
22 numprocs=1
23
24 Putprocessstdoutoutputinthisfile
25 stdout_logfile=/var/log/celery/picha_worker.log
26
27 Putprocessstderroutputinthisfile
28 stderr_logfile=/var/log/celery/picha_worker.log
29
30 Iftrue,thisprogramwillstartautomaticallywhensupervisordisstart
31 ed
32 autostart=true
33
34 Maybeoneoffalse,unexpected,ortrue.Iffalse,theprocesswillnev
35 er
36 beautorestarted.Ifunexpected,theprocesswillberestartwhenthepr
37 ogram
38 exitswithanexitcodethatisnotoneoftheexitcodesassociatedwit
39 hthis
40 processconfiguration(seeexitcodes).Iftrue,theprocesswillbe
41 unconditionallyrestartedwhenitexits,withoutregardtoitsexitcod
42 e.
43 autorestart=true
44
45 Thetotalnumberofsecondswhichtheprogramneedstostayrunningafte
46 r
47 astartuptoconsiderthestartsuccessful.
48 startsecs=10
49
50 Needtowaitforcurrentlyexecutingtaskstofinishatshutdown.
51 Increasethisifyouhaveverylongrunningtasks.

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 13/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

52 stopwaitsecs=600

WhenresortingtosendSIGKILLtotheprogramtoterminateit
sendSIGKILLtoitswholeprocessgroupinstead,
takingcareofitschildrenaswell.
killasgroup=true

ifyourbrokerissupervised,setitspriorityhigher
soitstartsfirst
priority=998

Celery Scheduler: picha_celerybeat.conf

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 14/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

1 ================================
2 celerybeatsupervisorexample
3 ================================
4
5 thenameofyoursupervisordprogram
6 [program:pichacelerybeat]
7
8 Setfullpathtoceleryprogramifusingvirtualenv
9 command=/home/mosh/.virtualenvs/picha/bin/celerybeatApichaloglevel=I
10 NFO
11
12 ThedirectorytoyourDjangoproject
13 directory=/home/mosh/sites/picha
14
15 Ifsupervisordisrunastherootuser,switchuserstothisUNIXusera
16 ccount
17 beforedoinganyprocessing.
18 user=mosh
19
20 Supervisorwillstartasmanyinstancesofthisprogramasnamedbynump
21 rocs
22 numprocs=1
23
24 Putprocessstdoutoutputinthisfile
25 stdout_logfile=/var/log/celery/picha_beat.log
26
27 Putprocessstderroutputinthisfile
28 stderr_logfile=/var/log/celery/picha_beat.log
29
30 Iftrue,thisprogramwillstartautomaticallywhensupervisordisstart
31 ed
32 autostart=true
33
34 Maybeoneoffalse,unexpected,ortrue.Iffalse,theprocesswillnev
35 er
36 beautorestarted.Ifunexpected,theprocesswillberestartwhenthepr
37 ogram
38 exitswithanexitcodethatisnotoneoftheexitcodesassociatedwit
39 hthis
40 processconfiguration(seeexitcodes).Iftrue,theprocesswillbe
41 unconditionallyrestartedwhenitexits,withoutregardtoitsexitcod
42 e.
43 autorestart=true

Thetotalnumberofsecondswhichtheprogramneedstostayrunningafte
r
astartuptoconsiderthestartsuccessful.
startsecs=10

ifyourbrokerissupervised,setitspriorityhigher
soitstartsfirst

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 15/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

priority=999

1 $touch/var/log/celery/picha_worker.log
2 $touch/var/log/celery/picha_beat.log

pichacelery
pichacelerybeat

1 $sudosupervisorctlreread
2 $sudosupervisorctlupdate

pichacelery

1 $sudosupervisorctlstoppichacelery
2 $sudosupervisorctlstartpichacelery
3 $sudosupervisorctlstatuspichacelery

Final Tips

Next steps

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 16/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

reread update

See an error in this post? Please submit a pull request on Github

(https://github.com/realpython/realpython-blog).

Want to learn more? Download the Real Python course.

Working with Large Excel Files in Pandas (/blog/python/working-with-large-excel- les-in-pandas/)

Python, Ruby, and Golang: A Web Service Application Comparison (/blog/python/python-ruby-and-golang-a-web-Service-

application-comparison/)

Comments
Categories

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 17/18
01/09/2016 Asynchronous Tasks with Django and Celery - Real Python

https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/ 18/18