Django’s database layer provides various ways to help developers get the most\nout of their databases. This document gathers together links to the relevant\ndocumentation, and adds various tips, organized under a number of headings that\noutline the steps to take when attempting to optimize your database usage.
\nAs general programming practice, this goes without saying. Find out what\nqueries you are doing and what they are costing you.\nUse QuerySet.explain() to understand how specific QuerySets are\nexecuted by your database. You may also want to use an external project like\ndjango-debug-toolbar, or a tool that monitors your database directly.
Remember that you may be optimizing for speed or memory or both, depending on\nyour requirements. Sometimes optimizing for one will be detrimental to the\nother, but sometimes they will help each other. Also, work that is done by the\ndatabase process might not have the same cost (to you) as the same amount of\nwork done in your Python process. It is up to you to decide what your\npriorities are, where the balance must lie, and profile all of these as required\nsince this will depend on your application and server.
\nWith everything that follows, remember to profile after every change to ensure\nthat the change is a benefit, and a big enough benefit given the decrease in\nreadability of your code. All of the suggestions below come with the caveat\nthat in your circumstances the general principle might not apply, or might even\nbe reversed.
\n…including:
\nIndexes. This is a number one priority, after you have determined from\nprofiling what indexes should be added. Use\nMeta.indexes or\nField.db_index to add these from\nDjango. Consider adding indexes to fields that you frequently query using\nfilter(),\nexclude(),\norder_by(), etc. as indexes may help\nto speed up lookups. Note that determining the best indexes is a complex\ndatabase-dependent topic that will depend on your particular application.\nThe overhead of maintaining an index may outweigh any gains in query speed.
Appropriate use of field types.
We will assume you have done the things listed above. The rest of this document\nfocuses on how to use Django in such a way that you are not doing unnecessary\nwork. This document also does not address other optimization techniques that\napply to all expensive operations, such as general purpose caching.
\nQuerySetsLink to this heading#Understanding QuerySets is vital to getting good\nperformance with simple code. In particular:
\nQuerySet evaluationLink to this heading#To avoid performance problems, it is important to understand:
\nthat QuerySets are lazy.
when they are evaluated.
As well as caching of the whole QuerySet, there is caching of the result of\nattributes on ORM objects. In general, attributes that are not callable will be\ncached. For example, assuming the example blog models:
>>> entry = Entry.objects.get(id=1)\n>>> entry.blog # Blog object is retrieved at this point\n>>> entry.blog # cached version, no DB access\nBut in general, callable attributes cause DB lookups every time:
\n>>> entry = Entry.objects.get(id=1)\n>>> entry.authors.all() # query performed\n>>> entry.authors.all() # query performed again\nBe careful when reading template code - the template system does not allow use\nof parentheses, but will call callables automatically, hiding the above\ndistinction.
\nBe careful with your own custom properties - it is up to you to implement\ncaching when required, for example using the\ncached_property decorator.
with template tagLink to this heading#To make use of the caching behavior of QuerySet, you may need to use the\nwith template tag.
iterator()Link to this heading#When you have a lot of objects, the caching behavior of the QuerySet can\ncause a large amount of memory to be used. In this case,\niterator() may help.
explain()Link to this heading#QuerySet.explain() gives you detailed information about how the database\nexecutes a query, including indexes and joins that are used. These details may\nhelp you find queries that could be rewritten more efficiently, or identify\nindexes that could be added to improve performance.
For instance:
\nAt the most basic level, use filter and exclude to do\nfiltering in the database.
Use F expressions to filter\nbased on other fields within the same model.
If these aren’t enough to generate the SQL you need:
\nRawSQLLink to this heading#A less portable but more powerful method is the\nRawSQL expression, which allows some SQL\nto be explicitly added to the query. If that still isn’t powerful enough:
Write your own custom SQL to retrieve data or populate models. Use django.db.connection.queries to find out what Django\nis writing for you and start from there.
There are two reasons to use a column with\nunique or\ndb_index when using\nget() to retrieve individual objects.\nFirst, the query will be quicker because of the underlying database index.\nAlso, the query could run much slower if multiple objects match the lookup;\nhaving a unique constraint on the column guarantees this will never happen.
So using the example blog models:
\n>>> entry = Entry.objects.get(id=10)\nwill be quicker than:
\n>>> entry = Entry.objects.get(headline="News Item Title")\nbecause id is indexed by the database and is guaranteed to be unique.
Doing the following is potentially quite slow:
\n>>> entry = Entry.objects.get(headline__startswith="News")\nFirst of all, headline is not indexed, which will make the underlying\ndatabase fetch slower.
Second, the lookup doesn’t guarantee that only one object will be returned.\nIf the query matches more than one object, it will retrieve and transfer all of\nthem from the database. This penalty could be substantial if hundreds or\nthousands of records are returned. The penalty will be compounded if the\ndatabase lives on a separate server, where network overhead and latency also\nplay a factor.
\nHitting the database multiple times for different parts of a single ‘set’ of\ndata that you will need all parts of is, in general, less efficient than\nretrieving it all in one query. This is particularly important if you have a\nquery that is executed in a loop, and could therefore end up doing many database\nqueries, when only one was needed. So:
\n\nQuerySet.values() and values_list()Link to this heading#When you only want a dict or list of values, and don’t need ORM model\nobjects, make appropriate usage of\nvalues().\nThese can be useful for replacing model objects in template code - as long as\nthe dicts you supply have the same attributes as those used in the template,\nyou are fine.
QuerySet.defer() and only()Link to this heading#Use defer() and\nonly() if there are database columns\nyou know that you won’t need (or won’t need in most cases) to avoid loading\nthem. Note that if you do use them, the ORM will have to go and get them in\na separate query, making this a pessimization if you use it inappropriately.
Don’t be too aggressive in deferring fields without profiling as the database\nhas to read most of the non-text, non-VARCHAR data from the disk for a\nsingle row in the results, even if it ends up only using a few columns. The\ndefer() and only() methods are most useful when you can avoid loading a\nlot of text data or for fields that might take a lot of processing to convert\nback to Python. As always, profile first, then optimize.
QuerySet.contains(obj)Link to this heading#…if you only want to find out if obj is in the queryset, rather than\nif obj in queryset.
QuerySet.count()Link to this heading#…if you only want the count, rather than doing len(queryset).
QuerySet.exists()Link to this heading#…if you only want to find out if at least one result exists, rather than if\nqueryset.
But:
\ncontains(), count(), and exists()Link to this heading#If you are going to need other data from the QuerySet, evaluate it immediately.
\nFor example, assuming a Group model that has a many-to-many relation to\nUser, the following code is optimal:
members = group.members.all()\n\nif display_group_members:\n if members:\n if current_user in members:\n print("You and", len(members) - 1, "other users are members of this group.")\n else:\n print("There are", len(members), "members in this group.")\n\n for member in members:\n print(member.username)\n else:\n print("There are no members in this group.")\nIt is optimal because:
\nSince QuerySets are lazy, this does no database queries if\ndisplay_group_members is False.
Storing group.members.all() in the members variable allows its\nresult cache to be reused.
The line if members: causes QuerySet.__bool__() to be called, which\ncauses the group.members.all() query to be run on the database. If there\naren’t any results, it will return False, otherwise True.
The line if current_user in members: checks if the user is in the result\ncache, so no additional database queries are issued.
The use of len(members) calls QuerySet.__len__(), reusing the result\ncache, so again, no database queries are issued.
The for member loop iterates over the result cache.
In total, this code does either one or zero database queries. The only\ndeliberate optimization performed is using the members variable. Using\nQuerySet.exists() for the if, QuerySet.contains() for the in,\nor QuerySet.count() for the count would each cause additional queries.
QuerySet.update() and delete()Link to this heading#Rather than retrieve a load of objects, set some values, and save them\nindividual, use a bulk SQL UPDATE statement, via QuerySet.update(). Similarly, do bulk deletes where possible.
\nNote, however, that these bulk update methods cannot call the save() or\ndelete() methods of individual instances, which means that any custom\nbehavior you have added for these methods will not be executed, including\nanything driven from the normal database object signals.
If you only need a foreign key value, use the foreign key value that is already on\nthe object you’ve got, rather than getting the whole related object and taking\nits primary key. i.e. do:
\nentry.blog_id\n대신:
\nentry.blog.id\nOrdering is not free; each field to order by is an operation the database must\nperform. If a model has a default ordering (Meta.ordering) and you don’t need it, remove\nit on a QuerySet by calling\norder_by() with no parameters.
Adding an index to your database may help to improve ordering performance.
\nUse bulk methods to reduce the number of SQL statements.
\nWhen creating objects, where possible, use the\nbulk_create() method to reduce the\nnumber of SQL queries. For example:
Entry.objects.bulk_create(\n [\n Entry(headline="This is a test"),\n Entry(headline="This is only a test"),\n ]\n)\n…is preferable to:
\nEntry.objects.create(headline="This is a test")\nEntry.objects.create(headline="This is only a test")\nNote that there are a number of caveats to this method, so make sure it’s appropriate\nfor your use case.
When updating objects, where possible, use the\nbulk_update() method to reduce the\nnumber of SQL queries. Given a list or queryset of objects:
entries = Entry.objects.bulk_create(\n [\n Entry(headline="This is a test"),\n Entry(headline="This is only a test"),\n ]\n)\nThe following example:
\nentries[0].headline = "This is not a test"\nentries[1].headline = "This is no longer a test"\nEntry.objects.bulk_update(entries, ["headline"])\n…is preferable to:
\nentries[0].headline = "This is not a test"\nentries[0].save()\nentries[1].headline = "This is no longer a test"\nentries[1].save()\nNote that there are a number of caveats to this method, so make sure it’s appropriate\nfor your use case.
When inserting objects into ManyToManyFields, use\nadd() with multiple\nobjects to reduce the number of SQL queries. For example:
my_band.members.add(me, my_friend)\n…is preferable to:
\nmy_band.members.add(me)\nmy_band.members.add(my_friend)\n…where Band and Artist are models with a many-to-many relationship.
When inserting different pairs of objects into\nManyToManyField or when the custom\nthrough table is defined, use\nbulk_create() method to reduce the\nnumber of SQL queries. For example:
PizzaToppingRelationship = Pizza.toppings.through\nPizzaToppingRelationship.objects.bulk_create(\n [\n PizzaToppingRelationship(pizza=my_pizza, topping=pepperoni),\n PizzaToppingRelationship(pizza=your_pizza, topping=pepperoni),\n PizzaToppingRelationship(pizza=your_pizza, topping=mushroom),\n ],\n ignore_conflicts=True,\n)\n…is preferable to:
\nmy_pizza.toppings.add(pepperoni)\nyour_pizza.toppings.add(pepperoni, mushroom)\n…where Pizza and Topping have a many-to-many relationship. Note that\nthere are a number of caveats to this method, so make sure it’s appropriate\nfor your use case.
When removing objects from ManyToManyFields, use\nremove() with multiple\nobjects to reduce the number of SQL queries. For example:
my_band.members.remove(me, my_friend)\n…is preferable to:
\nmy_band.members.remove(me)\nmy_band.members.remove(my_friend)\n…where Band and Artist are models with a many-to-many relationship.
When removing different pairs of objects from ManyToManyFields, use\ndelete() on a\nQ expression with multiple\nthrough model instances to reduce\nthe number of SQL queries. For example:
from django.db.models import Q\n\nPizzaToppingRelationship = Pizza.toppings.through\nPizzaToppingRelationship.objects.filter(\n Q(pizza=my_pizza, topping=pepperoni)\n | Q(pizza=your_pizza, topping=pepperoni)\n | Q(pizza=your_pizza, topping=mushroom)\n).delete()\n…is preferable to:
\nmy_pizza.toppings.remove(pepperoni)\nyour_pizza.toppings.remove(pepperoni, mushroom)\n…where Pizza and Topping have a many-to-many relationship.