Django offers a wide variety of built-in lookups for\nfiltering (for example, exact and icontains). This documentation\nexplains how to write custom lookups and how to alter the working of existing\nlookups. For the API references of lookups, see the Lookup API reference.
Let’s start with a simple custom lookup. We will write a custom lookup ne\nwhich works opposite to exact. Author.objects.filter(name__ne='Jack')\nwill translate to the SQL:
"author"."name" <> 'Jack'\nThis SQL is backend independent, so we don’t need to worry about different\ndatabases.
\nThere are two steps to making this work. Firstly we need to implement the\nlookup, then we need to tell Django about it. The implementation is quite\nstraightforward:
\nfrom django.db.models import Lookup\n\nclass NotEqual(Lookup):\n lookup_name = 'ne'\n\n def as_sql(self, compiler, connection):\n lhs, lhs_params = self.process_lhs(compiler, connection)\n rhs, rhs_params = self.process_rhs(compiler, connection)\n params = lhs_params + rhs_params\n return '%s <> %s' % (lhs, rhs), params\nTo register the NotEqual lookup we will just need to call\nregister_lookup on the field class we want the lookup to be available. In\nthis case, the lookup makes sense on all Field subclasses, so we register\nit with Field directly:
from django.db.models.fields import Field\nField.register_lookup(NotEqual)\nLookup registration can also be done using a decorator pattern:
\nfrom django.db.models.fields import Field\n\n@Field.register_lookup\nclass NotEqualLookup(Lookup):\n # ...\nWe can now use foo__ne for any field foo. You will need to ensure that\nthis registration happens before you try to create any querysets using it. You\ncould place the implementation in a models.py file, or register the lookup\nin the ready() method of an AppConfig.
Taking a closer look at the implementation, the first required attribute is\nlookup_name. This allows the ORM to understand how to interpret name__ne\nand use NotEqual to generate the SQL. By convention, these names are always\nlowercase strings containing only letters, but the only hard requirement is\nthat it must not contain the string __.
We then need to define the as_sql method. This takes a SQLCompiler\nobject, called compiler, and the active database connection.\nSQLCompiler objects are not documented, but the only thing we need to know\nabout them is that they have a compile() method which returns a tuple\ncontaining an SQL string, and the parameters to be interpolated into that\nstring. In most cases, you don’t need to use it directly and can pass it on to\nprocess_lhs() and process_rhs().
A Lookup works against two values, lhs and rhs, standing for\nleft-hand side and right-hand side. The left-hand side is usually a field\nreference, but it can be anything implementing the query expression API. The right-hand is the value given by the user. In the\nexample Author.objects.filter(name__ne='Jack'), the left-hand side is a\nreference to the name field of the Author model, and 'Jack' is the\nright-hand side.
We call process_lhs and process_rhs to convert them into the values we\nneed for SQL using the compiler object described before. These methods\nreturn tuples containing some SQL and the parameters to be interpolated into\nthat SQL, just as we need to return from our as_sql method. In the above\nexample, process_lhs returns ('"author"."name"', []) and\nprocess_rhs returns ('"%s"', ['Jack']). In this example there were no\nparameters for the left hand side, but this would depend on the object we have,\nso we still need to include them in the parameters we return.
Finally we combine the parts into an SQL expression with <>, and supply all\nthe parameters for the query. We then return a tuple containing the generated\nSQL string and the parameters.
The custom lookup above is great, but in some cases you may want to be able to\nchain lookups together. For example, let’s suppose we are building an\napplication where we want to make use of the abs() operator.\nWe have an Experiment model which records a start value, end value, and the\nchange (start - end). We would like to find all experiments where the change\nwas equal to a certain amount (Experiment.objects.filter(change__abs=27)),\nor where it did not exceed a certain amount\n(Experiment.objects.filter(change__abs__lt=27)).
We will start by writing an AbsoluteValue transformer. This will use the SQL\nfunction ABS() to transform the value before comparison:
from django.db.models import Transform\n\nclass AbsoluteValue(Transform):\n lookup_name = 'abs'\n function = 'ABS'\nNext, let’s register it for IntegerField:
from django.db.models import IntegerField\nIntegerField.register_lookup(AbsoluteValue)\nWe can now run the queries we had before.\nExperiment.objects.filter(change__abs=27) will generate the following SQL:
SELECT ... WHERE ABS("experiments"."change") = 27\nBy using Transform instead of Lookup it means we are able to chain\nfurther lookups afterwards. So\nExperiment.objects.filter(change__abs__lt=27) will generate the following\nSQL:
SELECT ... WHERE ABS("experiments"."change") < 27\nNote that in case there is no other lookup specified, Django interprets\nchange__abs=27 as change__abs__exact=27.
When looking for which lookups are allowable after the Transform has been\napplied, Django uses the output_field attribute. We didn’t need to specify\nthis here as it didn’t change, but supposing we were applying AbsoluteValue\nto some field which represents a more complex type (for example a point\nrelative to an origin, or a complex number) then we may have wanted to specify\nthat the transform returns a FloatField type for further lookups. This can\nbe done by adding an output_field attribute to the transform:
from django.db.models import FloatField, Transform\n\nclass AbsoluteValue(Transform):\n lookup_name = 'abs'\n function = 'ABS'\n\n @property\n def output_field(self):\n return FloatField()\nThis ensures that further lookups like abs__lte behave as they would for\na FloatField.
abs__lt lookupLink to this heading#When using the above written abs lookup, the SQL produced will not use\nindexes efficiently in some cases. In particular, when we use\nchange__abs__lt=27, this is equivalent to change__gt=-27 AND\nchange__lt=27. (For the lte case we could use the SQL BETWEEN).
So we would like Experiment.objects.filter(change__abs__lt=27) to generate\nthe following SQL:
SELECT .. WHERE "experiments"."change" < 27 AND "experiments"."change" > -27\nThe implementation is:
\nfrom django.db.models import Lookup\n\nclass AbsoluteValueLessThan(Lookup):\n lookup_name = 'lt'\n\n def as_sql(self, compiler, connection):\n lhs, lhs_params = compiler.compile(self.lhs.lhs)\n rhs, rhs_params = self.process_rhs(compiler, connection)\n params = lhs_params + rhs_params + lhs_params + rhs_params\n return '%s < %s AND %s > -%s' % (lhs, rhs, lhs, rhs), params\n\nAbsoluteValue.register_lookup(AbsoluteValueLessThan)\nThere are a couple of notable things going on. First, AbsoluteValueLessThan\nisn’t calling process_lhs(). Instead it skips the transformation of the\nlhs done by AbsoluteValue and uses the original lhs. That is, we\nwant to get "experiments"."change" not ABS("experiments"."change").\nReferring directly to self.lhs.lhs is safe as AbsoluteValueLessThan\ncan be accessed only from the AbsoluteValue lookup, that is the lhs\nis always an instance of AbsoluteValue.
Notice also that as both sides are used multiple times in the query the params\nneed to contain lhs_params and rhs_params multiple times.
The final query does the inversion (27 to -27) directly in the\ndatabase. The reason for doing this is that if the self.rhs is something else\nthan a plain integer value (for example an F() reference) we can’t do the\ntransformations in Python.
The AbsoluteValue example we discussed previously is a transformation which\napplies to the left-hand side of the lookup. There may be some cases where you\nwant the transformation to be applied to both the left-hand side and the\nright-hand side. For instance, if you want to filter a queryset based on the\nequality of the left and right-hand side insensitively to some SQL function.
Let’s examine the simple example of case-insensitive transformation here. This\ntransformation isn’t very useful in practice as Django already comes with a bunch\nof built-in case-insensitive lookups, but it will be a nice demonstration of\nbilateral transformations in a database-agnostic way.
\nWe define an UpperCase transformer which uses the SQL function UPPER() to\ntransform the values before comparison. We define\nbilateral = True to indicate that\nthis transformation should apply to both lhs and rhs:
from django.db.models import Transform\n\nclass UpperCase(Transform):\n lookup_name = 'upper'\n function = 'UPPER'\n bilateral = True\nNext, let’s register it:
\nfrom django.db.models import CharField, TextField\nCharField.register_lookup(UpperCase)\nTextField.register_lookup(UpperCase)\nNow, the queryset Author.objects.filter(name__upper="doe") will generate a case\ninsensitive query like this:
SELECT ... WHERE UPPER("author"."name") = UPPER('doe')\nSometimes different database vendors require different SQL for the same\noperation. For this example we will rewrite a custom implementation for\nMySQL for the NotEqual operator. Instead of <> we will be using !=\noperator. (Note that in reality almost all databases support both, including\nall the official databases supported by Django).
We can change the behavior on a specific backend by creating a subclass of\nNotEqual with an as_mysql method:
class MySQLNotEqual(NotEqual):\n def as_mysql(self, compiler, connection):\n lhs, lhs_params = self.process_lhs(compiler, connection)\n rhs, rhs_params = self.process_rhs(compiler, connection)\n params = lhs_params + rhs_params\n return '%s != %s' % (lhs, rhs), params\n\nField.register_lookup(MySQLNotEqual)\nWe can then register it with Field. It takes the place of the original\nNotEqual class as it has the same lookup_name.
When compiling a query, Django first looks for as_%s % connection.vendor\nmethods, and then falls back to as_sql. The vendor names for the in-built\nbackends are sqlite, postgresql, oracle and mysql.
In some cases you may wish to dynamically change which Transform or\nLookup is returned based on the name passed in, rather than fixing it. As\nan example, you could have a field which stores coordinates or an arbitrary\ndimension, and wish to allow a syntax like .filter(coords__x7=4) to return\nthe objects where the 7th coordinate has value 4. In order to do this, you\nwould override get_lookup with something like:
class CoordinatesField(Field):\n def get_lookup(self, lookup_name):\n if lookup_name.startswith('x'):\n try:\n dimension = int(lookup_name[1:])\n except ValueError:\n pass\n else:\n return get_coordinate_lookup(dimension)\n return super().get_lookup(lookup_name)\nYou would then define get_coordinate_lookup appropriately to return a\nLookup subclass which handles the relevant value of dimension.
There is a similarly named method called get_transform(). get_lookup()\nshould always return a Lookup subclass, and get_transform() a\nTransform subclass. It is important to remember that Transform\nobjects can be further filtered on, and Lookup objects cannot.
When filtering, if there is only one lookup name remaining to be resolved, we\nwill look for a Lookup. If there are multiple names, it will look for a\nTransform. In the situation where there is only one name and a Lookup\nis not found, we look for a Transform and then the exact lookup on that\nTransform. All call sequences always end with a Lookup. To clarify:
.filter(myfield__mylookup) will call myfield.get_lookup('mylookup').
.filter(myfield__mytransform__mylookup) will call\nmyfield.get_transform('mytransform'), and then\nmytransform.get_lookup('mylookup').
.filter(myfield__mytransform) will first call\nmyfield.get_lookup('mytransform'), which will fail, so it will fall back\nto calling myfield.get_transform('mytransform') and then\nmytransform.get_lookup('exact').