For a web application, there’s often more than just turning HTTP requests into\nHTTP responses. For some functionality, it may be beneficial to run code\noutside the request-response cycle.
\nThat’s where background Tasks come in.
\nBackground Tasks can offload work to be run outside the request-response cycle,\nto be run elsewhere, potentially at a later date. This keeps requests fast,\nreduces latency, and improves the user experience. For example, a user\nshouldn’t have to wait for an email to send before their page finishes loading.
\nDjango’s Tasks framework makes it easy to define and enqueue such work. It\ndoes not provide a worker mechanism to run Tasks. The actual execution must be\nhandled by infrastructure outside Django, such as a separate process or\nservice. Given that, a task backend capable\nof executing tasks on that service should be evaluated and configured.
\nWhen work needs to be done in the background, Django creates a Task, which\nis stored in the Queue Store. This Task contains all the metadata needed to\nexecute it, as well as a unique identifier for Django to retrieve the result\nlater.
A Worker will look at the Queue Store for new Tasks to run. When a new Task is\nadded, a Worker claims the Task, executes it, and saves the status and result\nback to the Queue Store. These workers run outside the request-response\nlifecycle.
\nThe Task backend determines how and where Tasks are stored for execution and\nhow they are executed. Different Task backends have different characteristics\nand configuration options, which may impact the performance and reliability of\nyour application. Django comes with built-in backends, but these are for development and testing only.
\nDjango handles task definition, validation, queuing, and result handling, not\nexecution, so production setups need a backend or worker process that actually\nruns queued work. Relevant options are listed in the Community Ecosystem page.
\nTask backends are configured using the TASKS setting in your\nsettings file. Whilst most applications will only need a single backend,\nmultiple are supported.
This is the default backend if another is not specified in your settings file.\nThe ImmediateBackend runs enqueued Tasks immediately, rather than in\nthe background. This allows background Task functionality to be slowly added to\nan application, before the required infrastructure is available.
To use it, set BACKEND to\n"django.tasks.backends.immediate.ImmediateBackend":
TASKS = {"default": {"BACKEND": "django.tasks.backends.immediate.ImmediateBackend"}}\nThe ImmediateBackend may also be useful in tests, to bypass the need\nto run a real background worker in your tests.
The DummyBackend doesn’t execute enqueued Tasks at all, instead\nstoring results for later use. Task results will forever remain in the\nREADY state.
This backend is not intended for use in production - it is provided as a\nconvenience that can be used during development and testing.
\nTo use it, set BACKEND to\n"django.tasks.backends.dummy.DummyBackend":
TASKS = {"default": {"BACKEND": "django.tasks.backends.dummy.DummyBackend"}}\nThe results for enqueued Tasks can be retrieved from the backend’s\nresults attribute:
>>> from django.tasks import default_task_backend\n>>> my_task.enqueue()\n>>> len(default_task_backend.results)\n1\nStored results can be cleared using the\nclear() method:
>>> default_task_backend.clear()\n>>> len(default_task_backend.results)\n0\nAs mentioned at the beginning of this section, Django includes backends\nsuitable for development and testing only. Production systems should rely on\nbackends that supply a worker process and a durable queue implementation.\nAvailable third-party backends are listed on the Community Ecosystem page and the Tasks\nframework grid from Django Packages.
\nTo use an external Task backend with Django, use the Python import path as the\nBACKEND of the TASKS setting, like so:
TASKS = {\n "default": {\n "BACKEND": "path.to.backend",\n }\n}\nA Task backend is a class that inherits\nBaseTaskBackend. At a minimum, it must\nimplement BaseTaskBackend.enqueue(). If you’re building your own\nbackend, you can use the built-in Task backends as reference implementations.\nYou’ll find the code in the django/tasks/backends/ directory of the\nDjango source.
Django has developing support for asynchronous Task backends.
\nBaseTaskBackend has async variants of all\nbase methods. By convention, the asynchronous versions of all methods are\nprefixed with a. The arguments for both variants are the same.
Backends can be retrieved using the task_backends connection handler:
from django.tasks import task_backends\n\ntask_backends["default"] # The default backend\ntask_backends["reserve"] # Another backend\nThe “default” backend is available as default_task_backend:
from django.tasks import default_task_backend\nTasks are defined using the django.tasks.task() decorator on a\nmodule-level function:
from django.core.mail import send_mail\nfrom django.tasks import task\n\n\n@task\ndef email_users(emails, subject, message):\n return send_mail(\n subject=subject, message=message, from_email=None, recipient_list=emails\n )\nThe return value of the decorator is a Task instance.
Task attributes can be customized via the @task\ndecorator arguments:
from django.core.mail import send_mail\nfrom django.tasks import task\n\n\n@task(priority=2, queue_name="emails")\ndef email_users(emails, subject, message):\n return send_mail(\n subject=subject, message=message, from_email=None, recipient_list=emails\n )\nBy convention, Tasks are defined in a tasks.py file, however this is not\nenforced.
Sometimes, the running Task may need to know context about how it was\nenqueued, and how it is being executed. This can be accessed by taking a\ncontext argument, which is an instance of\nTaskContext.
To receive the Task context as an argument to your Task function, pass\ntakes_context when defining it:
import logging\nfrom django.core.mail import send_mail\nfrom django.tasks import task\n\nlogger = logging.getLogger(__name__)\n\n\n@task(takes_context=True)\ndef email_users(context, emails, subject, message):\n logger.debug(\n f"Attempt {context.attempt} to send user email. Task result id: {context.task_result.id}."\n )\n return send_mail(\n subject=subject, message=message, from_email=None, recipient_list=emails\n )\nBefore enqueueing Tasks, it may be necessary to modify certain parameters of\nthe Task. For example, to give it a higher priority than it would normally.
\nA Task instance cannot be modified directly. Instead, a modified instance\ncan be created with the using() method, leaving the\noriginal as-is. For example:
>>> email_users.priority\n0\n>>> email_users.using(priority=10).priority\n10\nTo add the Task to the queue store, so it will be executed, call the\nenqueue() method on it. If the Task takes arguments,\nthese can be passed as-is. For example:
result = email_users.enqueue(\n emails=["user@example.com"],\n subject="You have a message",\n message="Hello there!",\n)\nThis returns a TaskResult, which can be used to retrieve\nthe result of the Task once it has finished executing.
To enqueue Tasks in an async context, aenqueue()\nis available as an async variant of enqueue().
Because both Task arguments and return values are serialized to JSON, they must\nbe JSON-serializable:
\n>>> process_data.enqueue(datetime.now())\nTraceback (most recent call last):\n...\nTypeError: Object of type datetime is not JSON serializable\nArguments must also be able to round-trip through a json.dumps()/\njson.loads() cycle without changing type. For example, consider this\nTask:
@task()\ndef double_dictionary(key):\n return {key: key * 2}\nWith the ImmediateBackend configured as the default backend:
>>> result = double_dictionary.enqueue((1, 2, 3))\n>>> result.status\nFAILED\n>>> result.errors[0].traceback\nTraceback (most recent call last):\n...\nTypeError: unhashable type: 'list'\nThe double_dictionary Task fails because after the JSON round-trip the\ntuple (1, 2, 3) becomes the list [1, 2, 3], which cannot be used as a\ndictionary key.
In general, complex objects such as model instances, or built-in types like\ndatetime and tuple cannot be used in Tasks without additional\nconversion.
For most backends, Tasks are run in a separate process, using a different\ndatabase connection. When using a transaction, without waiting for it to\ncommit, workers could start to process a Task which uses objects it can’t\naccess yet.
\nFor example, consider this simplified example:
\n@task\ndef my_task(thing_num):\n Thing.objects.get(num=thing_num)\n\n\nwith transaction.atomic():\n Thing.objects.create(num=1)\n my_task.enqueue(thing_num=1)\nTo prevent the scenario where my_task runs before the Thing is\ncommitted to the database, use transaction.on_commit(), binding all arguments to\nenqueue() via functools.partial():
from functools import partial\n\nfrom django.db import transaction\n\nwith transaction.atomic():\n Thing.objects.create(num=1)\n transaction.on_commit(partial(my_task.enqueue, thing_num=1))\nWhen enqueueing a Task, you receive a TaskResult,\nhowever it’s likely useful to retrieve the result from somewhere else (for\nexample another request or another Task).
Each TaskResult has a unique id, which can\nbe used to identify and retrieve the result once the code which enqueued the\nTask has finished.
The get_result() method can retrieve a result based on\nits id:
# Later, somewhere else...\nresult = email_users.get_result(result_id)\nTo retrieve a TaskResult, regardless of which kind of Task it was from,\nuse the get_result() method on the backend:
from django.tasks import default_task_backend\n\nresult = default_task_backend.get_result(result_id)\nTo retrieve results in an async context,\naget_result() is available as an async variant of\nget_result() on both the backend and Task.
Some backends, such as the built-in ImmediateBackend do not support\nget_result(). Calling get_result() on these backends will\nraise NotImplementedError.
A TaskResult contains the status of a Task’s execution at the point it was\nretrieved. If the Task finishes after get_result() is\ncalled, it will not update.
To refresh the values, call the django.tasks.TaskResult.refresh()\nmethod:
>>> result.status\nRUNNING\n>>> result.refresh() # or await result.arefresh()\n>>> result.status\nSUCCESSFUL\nIf your Task function returns something, it can be retrieved from the\ndjango.tasks.TaskResult.return_value attribute:
>>> result.status\nSUCCESSFUL\n>>> result.return_value\n42\nIf the Task has not finished executing, or has failed, ValueError is\nraised.
>>> result.status\nRUNNING\n>>> result.return_value\nTraceback (most recent call last):\n...\nValueError: Task has not finished yet\nIf the Task doesn’t succeed, and instead raises an exception, either as part of\nthe Task or as part of running it, the exception and traceback are saved to the\ndjango.tasks.TaskResult.errors list.
Each entry in errors is a TaskError containing\ninformation about error raised during the execution:
>>> result.errors[0].exception_class\n<class 'ValueError'>\nNote that this is just the type of exception, and contains no other values. The\ntraceback information is reduced to a string which you can use to help\ndebugging:
\n>>> result.errors[0].traceback\nTraceback (most recent call last):\n...\nTypeError: Object of type datetime is not JSON serializable\n